Prompt rather than classify? Zero- and few-shot evaluation of NLP models in the privacy domain

AI Quick Summary

This project seeks to explore prompt-based models (eg. T5 or T0) for solving NLP tasks in the privacy-domain. This is in contrast to non-prompt based NLP models such as BERT. We provide a set of NLP tasks in the privacy-domain and your task would be to transform one or more of these tasks into a prompt-based framework. Then, you will train prompt-based model(s) on these tasks on zero- and few-shot setting and evaluate whether a meaningful performance gain can be observed. BackgroundUsing prompt-based models we reformulate the classification tasks as natural text. Thereby, we prevent a cold start training and can gain from their ability to process language in general. Providing descriptions of the tasks itself along with the input allows us to apply them in low-resource scenarios like zero- or few-shot settings.When we consider the case of sentiment analysis, we can formulate a prompt where we expect "positive" or "negative" as a result:"Is this a positive or negative review: The movie was great."Goals and Expected ResultsLiterature review of existing prompt-based methodsSelection of 2-3 NLP tasks from a pool of provided tasksReformulate selected tasks as prompt-based variantsEvaluate zero-shot capabilities of pre-trained prompt-based models (eg. T0 or T5) on selected tasksFine-tune prompt-based models on selected tasks and evaluate in a few-shot settingCompare zero and few-shot performances with that of non-prompt-based NLP models (eg. BERT)LiteratureRaffel, C., Shazeer, N.M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P.J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. ArXiv, abs/1910.10683.Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L.A., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T.L., Raja, A., Dey, M., BARI, M., Xu, C., Thakker, U., Sharma, S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N.V., Datta, D., Chang, J., Jiang, M.T., Wang, H., Manica, M., Shen, S., Yong, Z., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Févry, T., Fries, J.A., Teehan, R., Biderman, S.R., Gao, L., Bers, T.G., Wolf, T., & Rush, A.M. (2021). Multitask Prompted Training Enables Zero-Shot Task Generalization. ArXiv, abs/2110.08207.

Project Idea Metadata

• Project Idea Name: Prompt rather than classify? Zero- and few-shot evaluation of NLP models in the privacy domain
• Date: 6/15/2022 12:34:39 PM
• Administrators:
  • Atreya Shankar

Project Idea Description

Background

Using prompt-based models we reformulate the classification tasks as natural text. Thereby, we prevent a cold start training and can gain from their ability to process language in general. Providing descriptions of the tasks itself along with the input allows us to apply them in low-resource scenarios like zero- or few-shot settings.

When we consider the case of sentiment analysis, we can formulate a prompt where we expect "positive" or "negative" as a result:

"Is this a positive or negative review: The movie was great."

Goals and Expected Results

1. Literature review of existing prompt-based methods
2. Selection of 2-3 NLP tasks from a pool of provided tasks
3. Reformulate selected tasks as prompt-based variants
4. Evaluate zero-shot capabilities of pre-trained prompt-based models (eg. T0 or T5) on selected tasks
5. Fine-tune prompt-based models on selected tasks and evaluate in a few-shot setting
6. Compare zero and few-shot performances with that of non-prompt-based NLP models (eg. BERT)

Literature

Raffel, C., Shazeer, N.M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P.J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. ArXiv, abs/1910.10683.

Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L.A., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T.L., Raja, A., Dey, M., BARI, M., Xu, C., Thakker, U., Sharma, S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N.V., Datta, D., Chang, J., Jiang, M.T., Wang, H., Manica, M., Shen, S., Yong, Z., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Févry, T., Fries, J.A., Teehan, R., Biderman, S.R., Gao, L., Bers, T.G., Wolf, T., & Rush, A.M. (2021). Multitask Prompted Training Enables Zero-Shot Task Generalization. ArXiv, abs/2110.08207.

This project seeks to explore prompt-based models (eg. T5 or T0) for solving NLP tasks in the privacy-domain. This is in contrast to non-prompt based NLP models such as BERT. We provide a set of NLP tasks in the privacy-domain and your task would be to transform one or more of these tasks into a prompt-based framework. Then, you will train prompt-based model(s) on these tasks on zero- and few-shot setting and evaluate whether a meaningful performance gain can be observed.