Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
Authors:
Hao Peng,
Qingqing Cao,
Jesse Dodge,
Matthew E. Peters,
Jared Fernandez,
Tom Sherborne,
Kyle Lo,
Sam Skjonsberg,
Emma Strubell,
Darrell Plessas,
Iz Beltagy,
Evan Pete Walsh,
Noah A. Smith,
Hannaneh Hajishirzi
Abstract:
Rising computational demands of modern natural language processing (NLP) systems have increased the barrier to entry for cutting-edge research while posing serious environmental concerns. Yet, progress on model efficiency has been impeded by practical challenges in model evaluation and comparison. For example, hardware is challenging to control due to disparate levels of accessibility across diffe…
▽ More
Rising computational demands of modern natural language processing (NLP) systems have increased the barrier to entry for cutting-edge research while posing serious environmental concerns. Yet, progress on model efficiency has been impeded by practical challenges in model evaluation and comparison. For example, hardware is challenging to control due to disparate levels of accessibility across different institutions. Moreover, improvements in metrics such as FLOPs often fail to translate to progress in real-world applications. In response, we introduce Pentathlon, a benchmark for holistic and realistic evaluation of model efficiency. Pentathlon focuses on inference, which accounts for a majority of the compute in a model's lifecycle. It offers a strictly-controlled hardware platform, and is designed to mirror real-world applications scenarios. It incorporates a suite of metrics that target different aspects of efficiency, including latency, throughput, memory overhead, and energy consumption. Pentathlon also comes with a software library that can be seamlessly integrated into any codebase and enable evaluation. As a standardized and centralized evaluation platform, Pentathlon can drastically reduce the workload to make fair and reproducible efficiency comparisons. While initially focused on natural language processing (NLP) models, Pentathlon is designed to allow flexible extension to other fields. We envision Pentathlon will stimulate algorithmic innovations in building efficient models, and foster an increased awareness of the social and environmental implications in the development of future-generation NLP models.
△ Less
Submitted 18 July, 2023;
originally announced July 2023.
On the Concrete Categories of Graphs
Authors:
George McRae,
Demitri Plessas,
Liam Rafferty
Abstract:
In the standard Category of Graphs, the graphs allow only one edge to be incident to any two vertices, not necessarily distinct, and the graph morphisms must map edges to edges and vertices to vertices while preserving incidence. We refer to these graph morphisms as Strict Morphisms. We relax the condition on the graphs allowing any number of edges to be incident to any two vertices, as well as re…
▽ More
In the standard Category of Graphs, the graphs allow only one edge to be incident to any two vertices, not necessarily distinct, and the graph morphisms must map edges to edges and vertices to vertices while preserving incidence. We refer to these graph morphisms as Strict Morphisms. We relax the condition on the graphs allowing any number of edges to be incident to any two vertices, as well as relaxing the condition on graph morphisms by allowing edges to be mapped to vertices, provided that incidence is still preserved. We call this broader graph category The Category of Conceptual Graphs, and define four other graph categories created by combinations of restrictions of the graph morphisms as well as restrictions on the allowed graphs. We investigate which Lawvere axioms for the category of Sets and Functions apply to each of these Categories of Graphs, as well as the other categorial constructions of free objects, projective objects, generators, and their categorial duals.
△ Less
Submitted 28 November, 2012;
originally announced November 2012.
