跳转至

文章背景与核心概要

图书馆和档案馆往往需要在人手紧缺和计算预算有限的情况下管理海量的藏书与文献资料，然而现有的标准基准测试却无法系统性地评估其核心的书目处理任务。为了弥补这一空白，本文介绍了 SHELF（Synthetic Harness for Evaluating LLM Fitness，用于评估大模型适应性的合成框架），这是一个基于 Python 的系统，旨在将带标签的分类体系、写作规范和生成预算转化为受控的基准数据与评估任务。

该框架的首个版本基于美国国会图书馆（Library of Congress）的词表，构建了包含 62,899 个由模型生成的文档，涵盖了分类、聚类、检索、对分类（pair classification）以及指令检索等任务。评估结果表明，模型在各项任务上的表现差异明显——主题分类取得了较高的准确率，而体裁-形式分类以及多项聚类/对分类任务依然极具挑战性。此外，尽管绝对得分无法直接映射到实际生产目录数据中，但在各项评估中模型之间的相对排名依然保持可靠。所有源代码和数据集均已在宽松的开源许可证下公开。

SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

Summary

Summary

Libraries and archives often manage massive collections with limited staff and computing budgets, yet standard benchmarks fail to systematically evaluate their core bibliographic tasks. To address this gap, SHELF (Synthetic Harness for Evaluating LLM Fitness) introduces a Python-based system designed to convert labeled taxonomies, writing specifications, and generation budgets into controlled benchmark data and evaluation tasks.

Libraries and archives often manage massive collections with limited staff and computing budgets, yet standard benchmarks fail to systematically evaluate their core bibliographic tasks. To address this gap, SHELF (Synthetic Harness for Evaluating LLM Fitness) introduces a Python-based system designed to convert labeled taxonomies, writing specifications, and generation budgets into controlled benchmark data and evaluation tasks.

The initial release features 62,899 model-generated documents built on Library of Congress vocabularies, spanning tasks such as classification, clustering, retrieval, pair classification, and instruction retrieval. Evaluation results show varying performance across tasks—subject classification achieves strong accuracy, while genre-form classification and several clustering/pair tasks remain challenging. Furthermore, while absolute scores do not directly map to production catalogue data, model rankings remain reliable across evaluations. All source code and datasets are publicly available under permissive licenses.

The initial release features 62,899 model-generated documents built on Library of Congress vocabularies, spanning tasks such as classification, clustering, retrieval, pair classification, and instruction retrieval. Evaluation results show varying performance across tasks—subject classification achieves strong accuracy, while genre-form classification and several clustering/pair tasks remain challenging. Furthermore, while absolute scores do not directly map to production catalogue data, model rankings remain reliable across evaluations. All source code and datasets are publicly available under permissive licenses.

Metadata

Metadata

Abstract

Abstract

Libraries and archives manage large collections with limited staff and computing budgets, yet common benchmarks do not systematically test their bibliographic work. They need to know which methods work for their tasks and what those methods require to run. SHELF, the Synthetic Harness for Evaluating LLM Fitness, addresses this gap. It is a Python system that turns labelled taxonomies, writing specifications, and a generation budget into controlled benchmark data and evaluation tasks. This first release contains 62,899 model-written documents based on Library of Congress vocabularies, with tasks for classification, clustering, retrieval, pair classification, and instruction retrieval. We compare TF, TF-IDF, BM25, popular encoders, and, on subject classification only, zero-shot decoders; each method appears only on tasks that support it. Subject classification reaches 0.8887, while genre-form classification reaches only 0.2605, and several pair and clustering tasks remain near chance. Sparse methods remain competitive on classification, while TF-IDF is the fastest measured arm in the subject timing experiment. SHELF also varies bibliographic facets independently and can generate new, verifiably unseen documents after a model's training cutoff. Comparisons with LCSHBench and Project Gutenberg show that model rankings transfer more reliably than absolute scores, but SHELF scores do not estimate accuracy on production catalogue data. We release all source code and data under permissive licenses on GitHub and Hugging Face.

Libraries and archives manage large collections with limited staff and computing budgets, yet common benchmarks do not systematically test their bibliographic work. They need to know which methods work for their tasks and what those methods require to run. SHELF, the Synthetic Harness for Evaluating LLM Fitness, addresses this gap. It is a Python system that turns labelled taxonomies, writing specifications, and a generation budget into controlled benchmark data and evaluation tasks. This first release contains 62,899 model-written documents based on Library of Congress vocabularies, with tasks for classification, clustering, retrieval, pair classification, and instruction retrieval. We compare TF, TF-IDF, BM25, popular encoders, and, on subject classification only, zero-shot decoders; each method appears only on tasks that support it. Subject classification reaches 0.8887, while genre-form classification reaches only 0.2605, and several pair and clustering tasks remain near chance. Sparse methods remain competitive on classification, while TF-IDF is the fastest measured arm in the subject timing experiment. SHELF also varies bibliographic facets independently and can generate new, verifiably unseen documents after a model's training cutoff. Comparisons with LCSHBench and Project Gutenberg show that model rankings transfer more reliably than absolute scores, but SHELF scores do not estimate accuracy on production catalogue data. We release all source code and data under permissive licenses on GitHub and Hugging Face.

License Information

License Information

license icon View License (Creative Commons Attribution 4.0 International)

license icon View License (Creative Commons Attribution 4.0 International)