Massw

<p align="center"> <i>Figure: A visualization of key ideas' evolution through time. Darker points indicate more recent work.</i> </p> <p align="center"> <a href="https://arxiv.org/abs/2406.06357"> <img src="https://img.shields.io/badge/arXiv-2406.06357-b31b1b.svg" alt="arXiv"> </a> <a href="http://creativecommons.org/publicdomain/zero/1.0/"> <img src="https://img.shields.io/badge/License-CC0_1.0-lightgrey.svg" alt="License: CC0-1.0"> </a> <a href="https://github.com/xingjian-zhang/massw/actions/workflows/pycodestyle.yml"> <img src="https://github.com/xingjian-zhang/massw/actions/workflows/pycodestyle.yml/badge.svg" alt="Pycodestyle"> </a> <a href="https://github.com/xingjian-zhang/massw/actions/workflows/pydocstyle.yml"> <img src="https://github.com/xingjian-zhang/massw/actions/workflows/pydocstyle.yml/badge.svg" alt="Pydocstyle"> </a> <a href="https://github.com/xingjian-zhang/massw/actions/workflows/pylint.yml"> <img src="https://github.com/xingjian-zhang/massw/actions/workflows/pylint.yml/badge.svg" alt="Pylint"> </a> </p> <!-- > This codebase contains the reproduction code for the paper "MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows". -->

MASSW is a comprehensive text dataset on Multi-Aspect Summarization of Scientific Workflows. MASSW includes more than 152,000 peer-reviewed publications from 17 leading computer science conferences spanning the past 50 years.

UPDATE: MASSW is now available on Hugging Face! See here for more details.

from datasets import load_dataset
ds = load_dataset("jimmyzxj/massw", "massw_data")
ds_metadata = load_dataset("jimmyzxj/massw", "massw_metadata")

Key features

• Structured scientific workflows. MASSW defines five core aspects of a scientific workflow -- context, key idea, method, outcome, and projected impact. These aspects align with the typical stages in scientific workflows identified in recent literature. Utilizing LLMs, we systematically extract and structure these five aspects from each publication.
• Large scale. MASSW contains the structured scientific workflows and meta-information from over 152,000 peer-reviewed publications, across 17 leading computer science conferences, and spanning the past 50 years.
• Accuracy. The coverage and accuracy of MASSW have been validated through comprehensive inspections and comparisons with human annotations and alternative methods.
• Rich benchmark tasks. MASSW facilitates multiple novel and benchmarkable machine learning tasks, such as idea generation and outcome prediction. It supports diverse tasks centered on predicting, recommending, and expanding key elements of a scientific workflow, serving as a benchmark for evaluating LLM agents' ability to navigate scientific research.

Core aspects of a scientific workflow

| Aspect | Definition | Example | |---------------|------------|---------| | Context <br> Ask questions, review literature (prior to study) | The status quo of related literature or reality which motivated this study. This could normally be a problem, a research question, or a research gap that has not been successfully addressed by previous work. | Making language models bigger does not inherently make them better at following a user's intent, as large models can generate outputs that are untruthful, toxic, or not helpful. | | Key Idea <br> Construct hypothesis (proposed in this study) | The main intellectual merit of this paper, often in comparison to the context. This could normally be a novel idea or solution proposed in this paper that distinguishes it from what’s already done in literature. | The authors propose InstructGPT, a method to align language models with user intent by fine-tuning GPT-3 using a combination of supervised learning with labeler demonstrations and reinforcement learning from human feedback. | | Method <br> Test hypothesis (after hypothesis construction) | The specific research method that investigates and validates the key idea. This could be an experimental setup, a theoretical framework, or other necessary validation methodology to implement and/or evaluate the key idea. | The authors evaluate the performance of InstructGPT by humans on a given prompt distribution and compare it with a much larger model GPT-3. | | Outcome <br> Interpret results, draw conclusion (after testing hypothesis) | The factual statement about the study output. This could be the experiment results and any other measurable outcome that has occurred. It marks whether the key hypothesis is testified or not. | InstructGPT, even with 100x fewer parameters, is preferred over GPT-3 in human evaluations. It shows improvements in truthfulness and reductions in toxic outputs with minimal performance regressions on public NLP datasets. | | Projected Impact <br> Future work (anticipated but not yet done) | The author-anticipated impact of the work on the field, and potential further research identified by the author that may improve or extend this study. | Fine-tuning with human feedback is a promising direction for aligning language models with human intent. |

Coverage

MASSW covers 17 leading computer science conferences, including:

• Artificial Intelligence: AAAI, IJCAI;
• Computer Vision: CVPR, ECCV, ICCV;
• Machine Learning: ICLR, ICML, NeurIPS, KDD;
• Natural Language Processing: ACL, EMNLP, NAACL;
• The Web & Information Retrieval: SIGIR, WWW;
• Databases: SIGMOD, VLDB;
• Interdisciplinary Areas: CHI.

Installation

pip install -r requirements.txt
pip install -e .

Basic Usage

Search for a specific publication by title:

from massw.data import load_massw

massw = load_massw()
massw.search("attention is all you need")

Output:

[Workflow(id='618cfba991e011333c932618', title='Cross-Attention is All You Need - Adapting Pretrained Transformers for Machine Translation.', abstract='We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross-attention when training from scratch. We conduct a series of experiments through fine-tuning a translation model on data where either the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that limiting fine-tuning in this manner yields cross-lingually aligned embeddings. The implications of this finding for researchers and practitioners include a mitigation of catastrophic forgetting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with reduced parameter storage overhead.', year=2021.0, venue='EMNLP', context='Cross-attention mechanism in the Transformer architecture has been of interest in the context of transfer learning for machine translation, and there exists previous studies of cross-attention when training from scratch.', key_idea='The authors explore the efficiency of fine-tuning only the cross-attention parameters in a translation model where either the source or target language has changed, as compared to fine-tuning all parameters (the entire translation model).', method='A series of experiments were conducted through fine-tuning a translation model on data where either the source or target language has changed.', outcome='The results revealed that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters. The process leads to cross-lingually aligned embeddings.', projected_impact='The findings can mitigate catastrophic forgetting, facilitate zero-shot translation, and allow extension of machine translation models to several new language pairs with reduced parameter storage overhead.', data={'id': '618cfba991e011333c932618', 'title': 'Cross-Attention is All You Need - Adapting Pretrained Transformers for Machine Translation.', 'abstract': 'We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross-attention when training from scratch. We conduct a series of experiments through fine-tuning a translation model on data where either the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that limiting fine-tuning in this manner yields cross-lingually aligned embeddings. The implications of this finding for researchers and practitioners include a mitigation of catastrophic forgetting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with reduced parameter storage overhead.', 'keywords': [], 'year': 2021, 'authors': [{'id': '64c1ec89688ef0ffd383e8b3', 'name': 'Mozhdeh Gheini', 'org': 'Univ Southern Calif, Informat Sci Inst, Los Angeles, CA 90007 USA', 'org_id': ''}, {'id': '5631ff8c45cedb3399f8b414', 'name': 'Xiang Ren', 'org': 'Univ Southern Calif, Informat Sci Inst, Los Angeles, CA 90007 USA', 'org_id': ''}, {'id': '5616d61c45ce1e5963b0b70e', 'name': 'Jonathan May', 'org': 'Univ Southern Calif, Informat Sci Inst, Los Angeles, CA 90007 USA', 'org_id': ''}], 'references': ['59ae3c152bbe271c4c71e900', '5e296ede3a55ac6ad1e44fd9', '5de240bd3a55ace1893738ef'], 'doi': '', 'venue_id': '5eba7087edb6e7d53c1009a5', 'n_citation': 38, 'venue': 'EMNLP', 'normali