CGFEV: Auditable LLM-Assisted Qualification of Configuration-Sensitive Manufacturing Evidence for Commercial Seating Design
DOI:
https://doi.org/10.4108/eetsis.14612Keywords:
evidence qualification, product configuration, large language models, intelligent manufacturing, automated compliance checking, provenanceAbstract
Engineering rules often rely on dimensions copied from supplier documents. A value may look correct yet belong to another chair option, measurement object, or source that lacks a required condition. Contract-Guided Frozen-Evidence Verification (CGFEV) freezes assigned files and target configuration, while a Rule Card states what must be established. An LLM writes typed propositions and locators; deterministic Gate v0.2 controls release. We evaluated four frozen routes on commercial seating. Holdout-A contained 48 family-isolated natural cases, including 47 with binary Gold; Holdout-B contained 20 controlled pairs. On Holdout-A, B1 released 4/8 Gold-ELIGIBLE cases and falsely admitted 6/39 Gold-NOT_ELIGIBLE cases. R0, L1, and the text-only V1 prompt variant recorded no observed false admissions and released 0/8 eligible cases. R0 completed 47/47 binary cases, whereas L1 and V1 ended in ERROR on 12 and 14 cases, leaving substantive decisions for 35/47 and 33/47. Every route missed all six eligible members and all four required direction changes in Holdout-B. All 88 formal V1 records had empty visual_assets fields, so these results provide no evidence about visual or multimodal qualification. Together, these results distinguish false admission, eligible release, and execution completion as separate evaluation dimensions. CGFEV provides an auditable admission boundary for configuration-sensitive evidence qualification.
References
[1] Sabin D, Weigel R. Product configuration frameworks: A survey. IEEE Intelligent Systems. 1998;13(4):42-49.
[2] Aldanondo M, Vareilles E. Configuration for mass customization: How to extend product configuration towards requirements and process configuration. Journal of Intelligent Manufacturing. 2008;19(5):521-535.
[3] Jiao J, Simpson TW, Siddique Z. Product family design and platform-based product development: A state-of-the-art review. Journal of Intelligent Manufacturing. 2007;18(1):5-29.
[4] Simpson TW. Product platform design and customization: Status and promise. Artificial Intelligence for Engineering Design, Analysis and Manufacturing. 2004;18(1):3-20.
[5] Opasjumruskit K, Schindler S, Peters D. Automatic data sheet information extraction for supporting model-based systems engineering. In: Luo Y, editor. Cooperative Design, Visualization, and Engineering. CDVE 2021. Lecture Notes in Computer Science. Vol 12983. Cham: Springer; 2021. p. 97-102.
[6] Harper CA, Cox J, Kohler C, Scerri A, Daniel R Jr, Groth P. SemEval-2021 Task 8: MeasEval-Extracting counts and measurements and their related contexts. In: Proceedings of the 15th International Workshop on Semantic Evaluation. 2021. p. 306-316.
[7] Göpfert J, Kuckertz P, Weinand JM, Kotzur L, Stolten D. Measurement extraction with natural language processing: A review. In: Findings of the Association for Computational Linguistics: EMNLP 2022. 2022. p. 2191-2215.
[8] Rizvi STR, Mercier D, Agne S, Erkel S, Dengel A, Ahmed S. Ontology-based information extraction from technical documents. In: Proceedings of the 10th International Conference on Agents and Artificial Intelligence. Vol 2. 2018. p. 493-500.
[9] Brinkmann A, Shraga R, Bizer C. ExtractGPT: Exploring the potential of large language models for product attribute value extraction. In: Information Integration and Web Intelligence. Lecture Notes in Computer Science. 2024. p. 38-52.
[10] Raza M, Jahangir Z, Riaz MB, Saeed MJ, Sattar MA. Industrial applications of large language models. Scientific Reports. 2025;15:13755.
[11] Xu Y, Li M, Cui L, Huang S, Wei F, Zhou M. LayoutLM: Pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2020. p. 1192-1200.
[12] Kim G, Hong T, Yim M, et al. OCR-free document understanding transformer. In: Computer Vision - ECCV 2022. Lecture Notes in Computer Science. Vol 13688. Cham: Springer; 2022. p. 498-517.
[13] Chen W, Wang H, Chen J, et al. TabFact: A large-scale dataset for table-based fact verification. In: Proceedings of the International Conference on Learning Representations. 2020.
[14] Eastman C, Lee JM, Jeong YS, Lee JK. Automatic rule-based checking of building designs. Automation in Construction. 2009;18(8):1011-1033.
[15] Ramonat M, Müller AW, Fay A. Towards automation of regulatory compliance checking in the product design phase. In: Proceedings of the 13th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management. 2021. p. 136-143.
[16] Wu J, Zhang J, Debs L. Model validation for automated building code compliance checking. In: Construction Research Congress 2022. Reston (VA): American Society of Civil Engineers; 2022. p. 640-650.
[17] Verhagen WJC, Bermell-Garcia P, van Dijk REC, Curran R. A critical review of knowledge-based engineering: An identification of research challenges. Advanced Engineering Informatics. 2012;26(1):5-15.
[18] Zhang J, El-Gohary NM. Automated information transformation for automated regulatory compliance checking in construction. Journal of Computing in Civil Engineering. 2015;29(4):B4015001.
[19] Solihin W, Eastman C. Classification of rules for automated BIM rule checking development. Automation in Construction. 2015;53:69-82.
[20] Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems. 2020;33. p. 9459-9474.
[21] Thorne J, Vlachos A, Christodoulopoulos C, Mittal A. FEVER: A large-scale dataset for fact extraction and verification. In: Proceedings of NAACL-HLT. 2018. p. 809-819.
[22] Barik AM, Hsu W, Lee ML. Time matters: An end-to-end solution for temporal claim verification. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2024. p. 657-664.
[23] Ge Z, Wu Y, Chin DWK, Lee RKW, Cao R. Resolving conflicting evidence in automated fact-checking: A study on retrieval-augmented LLMs. In: Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. 2025. p. 9656-9664.
[24] Es S, James J, Espinosa-Anke L, Schockaert S. RAGAs: Automated evaluation of retrieval augmented generation. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 2024. p. 150-158.
[25] Gao T, Yen H, Yu J, Chen D. Enabling large language models to generate text with citations. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. p. 6465-6488.
[26] Huang L, Yu W, Ma W, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems. 2025;43(2):1-55.
[27] Geifman Y, El-Yaniv R. Selective classification for deep neural networks. In: Advances in Neural Information Processing Systems. 2017;30. p. 4878-4887.
[28] Geifman Y, El-Yaniv R. SelectiveNet: A deep neural network with an integrated reject option. In: Proceedings of Machine Learning Research. 2019;97. p. 2151-2159.
[29] Guo C, Pleiss G, Sun Y, Weinberger KQ. On calibration of modern neural networks. In: Proceedings of Machine Learning Research. 2017;70. p. 1321-1330.
[30] Ovadia Y, Fertig E, Ren J, et al. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In: Advances in Neural Information Processing Systems. 2019;32. p. 13991-14002.
[31] Brinkmann A, Baumann N, Bizer C. Using LLMs for the extraction and normalization of product attribute values. In: Advances in Databases and Information Systems. Lecture Notes in Computer Science. 2024. p. 217-230.
[32] Geng S, Josifoski M, Peyrard M, West R. Grammar-constrained decoding for structured NLP tasks without finetuning. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. p. 10932-10952.
[33] Norheim JJ, Rebentisch E. Structuring natural language requirements with large language models. In: 2024 IEEE 32nd International Requirements Engineering Conference Workshops (REW). 2024. p. 68-71.
[34] Gebru T, Morgenstern J, Vecchione B, et al. Datasheets for datasets. Communications of the ACM. 2021;64(12):86-92.
[35] Mitchell M, Wu S, Zaldivar A, et al. Model cards for model reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency. 2019. p. 220-229.
[36] Moreau L, Missier P, editors. PROV-DM: The PROV data model. W3C Recommendation. World Wide Web Consortium; 2013.
[37] Zhang LL. Product configuration: A review of the state-of-the-art and future research. International Journal of Production Research. 2014;52(21):6381-6398.
[38] Wang RY, Strong DM. Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems. 1996;12(4):5-33.
[39] Strong DM, Lee YW, Wang RY. Data quality in context. Communications of the ACM. 1997;40(5):103-110.
[40] Serra F, Peralta V, Marotta A, Marcel P. Use of context in data quality management: A systematic literature review. Journal of Data and Information Quality. 2024;16(3):Article 19.
[41] Rebedea T, Dinu R, Sreedhar MN, Parisien C, Cohen J. NeMo Guardrails: A toolkit for controllable and safe LLM applications with programmable rails. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2023. p. 431-445.
[42] Scholak T, Schucher N, Bahdanau D. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. p. 9895-9901.
[43] Wu Y, Roesner F, Kohno T, Zhang N, Iqbal U. IsolateGPT: An execution isolation architecture for LLM-based agentic systems. In: Proceedings of the 2025 Network and Distributed System Security Symposium. 2025.
[44] Nuyts E, Bonduel M, Verstraeten R. Comparative analysis of approaches for automated compliance checking of construction data. Advanced Engineering Informatics. 2024;60:102443.
[45] Atanasova P, Simonsen JG, Lioma C, Augenstein I. Fact checking with insufficient evidence. Transactions of the Association for Computational Linguistics. 2022;10:746-763.
[46] Arnold M, Bellamy RKE, Hind M, et al. FactSheets: Increasing trust in AI services through supplier's declarations of conformity. IBM Journal of Research and Development. 2019;63(4/5):6:1-6:13.
[47] Cleland-Huang J, Berenbach B, Clark S, Settimi R, Romanova E. Best practices for automated traceability. Computer. 2007;40(6):27-35.
[48] Zhao L, Alhoshan W, Ferrari A, et al. Natural language processing for requirements engineering: A systematic mapping study. ACM Computing Surveys. 2022;54(3):Article 55.
[49] Mahmoud A, Niu N. On the role of semantics in automated requirements tracing. Requirements Engineering. 2015;20(3):281-300.
[50] Ali R, Dalpiaz F, Giorgini P. Reasoning with contextual requirements: Detecting inconsistency and conflicts. Information and Software Technology. 2013;55(1):35-57.
[51] Dianat I, Molenbroek J, Castellucci HI. A review of the methodology and applications of anthropometry in ergonomics and product design. Ergonomics. 2018;61(12):1696-1720.
[52] Manavis A, Minaoglou P, Efkolidis N, Kyratsis P. Digital customization for product design and manufacturing: A case study within the furniture industry. Electronics. 2024;13(13):2483.
[53] Dalipi F, Kurti A, Ferati M. APPEND: A blockchain-based model of digital product passport for furniture industry. In: Human Interaction and Emerging Technologies (IHIET-AI 2024): Artificial Intelligence and Future Applications. 2024;120. p. 95-103.
[54] Schuitemaker R, Xu X. Product traceability in manufacturing: A technical review. Procedia CIRP. 2020;93:700-705.
[55] Artstein R, Poesio M. Inter-coder agreement for computational linguistics. Computational Linguistics. 2008;34(4):555-596.
[56] Liang P, Bommasani R, Lee T, et al. Holistic evaluation of language models. Transactions on Machine Learning Research. 2023.
[57] Jacovi A, Caciularu A, Goldman O, Goldberg Y. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. p. 5075-5084.
[58] Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023;4(9):100804.
[59] Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proceedings of the National Academy of Sciences. 2018;115(11):2600-2606.
[60] Wilson EB. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association. 1927;22(158):209-212.
[61] McNemar Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika. 1947;12(2):153-157.
[62] Efron B. Bootstrap methods: Another look at the jackknife. The Annals of Statistics. 1979;7(1):1-26.
[63] Dauphin HumanDesign Group GmbH & Co. KG. Dynamic sitting in the office: Ergonomic technologies by Dauphin [Internet]. Offenhausen (Germany): Dauphin HumanDesign Group GmbH & Co. KG; [cited 2026 Sep 3]. Available from: https://www.dauphin.de/dauphin/de/englisch/ergonomics/mechanisms
[64] Sedus Stoll Aktiengesellschaft. black dot: Ergonomic swivel chair family [Internet]. Dogern (Germany): Sedus Stoll Aktiengesellschaft; [cited 2026 Sep 3]. Available from: https://www.sedus.com/en/products/chairs/black-dot
[65] Ho GTS, Tang YM, Tsang KY, Tang V, Chau KY. A blockchain-based system to enhance aircraft parts traceability and trackability for inventory management. Expert Systems with Applications. 2021;179:115101.
[66] Kuhn M, Franke J. Smart manufacturing traceability for automotive E/E systems enabled by event-driven microservice architecture. In: 2020 IEEE 11th International Conference on Mechanical and Intelligent Manufacturing Technologies. 2020. p. 142-148.
[67] Neelamkavil J, Pardasani A, Kernahan M. Traceability in medical devices design & manufacturing. In: Proceedings of the Canadian Engineering Education Association (CEEA). 2011.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Xu Huang, Chuanzhi Chen
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited.
