Hugging Face ecosystem | HF NLP course | 2. Using Hugging Face Transformers | Priority

Question 1

Code to get a tokenizer.

hugging-face tokenizers

Answer 1

from transformers import AutoTokenizer
checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

Source

Question 2

Code to send example input through a tokenizer with arguments.

hugging-face tokenizers

Answer 2

raw_inputs = [
    "I've been waiting for a HuggingFace course my whole life.",
    "I hate this so much!",
]
inputs = tokenizer(raw_inputs, padding=True, truncation=True, return_tensors="pt")
print(inputs)

Source

Question 3

Output format from a tokenizer.

hugging-face tokenizers

Answer 3

{'input_ids': tensor[[sentence1,…], 'attention_mask': tensor[[sentence1,…]}

Source

Question 4

Code to get a model (not for a specific task).

hugging-face transformers

Answer 4

from transformers import AutoModel
checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModel.from_pretrained(checkpoint)

hugging-face tokenizers

Question 5

What are the output dimensions of a Transformer module?

hugging-face transformers

Answer 5

batch size, sequence length, hidden size

Source

Question 6

Example code to feed the outputs of a tokenizer into a model.

hugging-face tokenizers transformers

Answer 6

outputs = model(**inputs)
print(outputs.last_hidden_state.shape)

Source

Question 7

Example code to get a model that will classify text. What will the output shape be?

hugging-face transformers

Answer 7

from transformers import AutoModelForSequenceClassification
checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
outputs = model(**inputs) # (batch_size, num_classes)

Source

Question 8

[page] Tokenizers: [page section] Loading and saving: [q] Code to use the AutoTokenizer class to grab the proper tokenizer class in the library based on the checkpoint name.

Answer 8

"from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(""bert-base-cased"")"

Source

Question 9

[page] Tokenizers:[page section] Loading and saving: [q] What are the 3 elements in the dict output of a tokenizer?

Answer 9

input_ids, token_type_ids, attention_mask

Source

Question 10

[page] Tokenizers: [page section] Loading and saving: [q] Saving a tokenizer.

Answer 10

tokenizer.save_pretrained(“directory_on_my_computer”)

Source

Question 11

[page] Handling multiple sequences: [page section] Models expect a batch of inputs: [q] Models expect ? sentences by default.

Answer 11

multiple

Source

Question 12

[page] Putting it all together: [page section]: Wrapping up: From tokenizer to model: [q] Write a code snippet that uses the tokenizer API to tokenize 2 sequences (using 3 arguments) and run them through a sequence classification model.

Answer 12

from transformers import AutoTokenizer, AutoModelForSequenceClassification
checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
sequences = ["I've been waiting for a HuggingFace course my whole life.", "So have I!"]					
tokens = tokenizer(sequences, padding=True, truncation=True, return_tensors="pt")
output = model(**tokens)

Source

Question 13

[page] Processing the data: [page section]: Loading a dataset from the Hub: [q] Code to download and cache the GLUE benchmark dataset from the Hugging Face hub.

Answer 13

from datasets import load_dataset
raw_datasets = load_dataset(""glue"")

Source

Question 14

[page] Processing the data: [page section] Loading a dataset from the Hub: [q] How to inspect the features of a dataset?

Answer 14

dataset.features

Source

Question 15

[page] Processing the data: [page section] Loading a dataset from the Hub: [q] How to tokenize all elements in a HF dataset?

Answer 15

"def tokenize_function(example):
	return tokenizer(…)
tokenized_dataset = dataset.map(tokenize_function)"

Source > video

Question 16

[page] Processing the data: [page section] Preprocessing a dataset: [q] What is the idiom to decode IDs to words?

Answer 16

tokenizer.convert_ids_to_tokens(inputs[“input_ids”])

Source

Question 17

[page] Processing the data: [page section] Dynamic padding: [q] What is a collate function? What is the default behavior?

Answer 17

The function that is responsible for putting together samples inside a batch. It’s an argument you can pass when you build a DataLoader, the default being a function that will just convert your samples to PyTorch tensors and concatenate them.

Source

Question 18

[page] Processing the data: [page section] Dynamic padding: [q] Example of how to create a collator?

Answer 18

“from transformers import DataCollatorWithPadding
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
batch = data_collator(samples)”

Source

Question 19

When not using the Trainer class, what does creating the DataLoader objects look like (using a collator for dynamic padding)?

Answer 19

"from torch.utils.data import DataLoader
from transformers import DataCollatorWithPadding 
data_collator = DataCoIIatorWithPadding(tokenizer) 
train_dataloader = DataLoader(
    tokenized_datasets[""train""], shuffle=True, batch_size=8, collate_fn=data_collator
)
eval_dataloader = DataLoader(
    tokenized_datasets[""validation""], batch_size=8, collate_fn=data_collator
)
for step, batch in enumerate(train_dataloader): 
    print(batch[ ""input_ids""].shape) 
    if step > 5: 
        break "

Source

Question 20

[page] Fine-tuning a model with the Trainer API: [page section] Training: [q] How to pass parameters to training?

Answer 20

“from transformers import TrainingArguments
training_args = TrainingArguments(““test-trainer””)”

Source

Question 21

[page] Fine-tuning a model with the Trainer API: [page section] Training: [q] How to instantiate a Trainer and start training, with evaluation per epoch?

Answer 21

"from transformers import Trainer
trainer = Trainer(
    model,
    training_args,
    train_dataset=tokenized_datasets[""train""],
    eval_dataset=tokenized_datasets[""validation""],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)
trainer.train()"

Source

Question 22

[page] Fine-tuning a model with the Trainer API: [page section] Evaluation: [q] What is the output of the trainer.predict() method?

Answer 22

named tuple with three fields: predictions, label_ids, and metrics

Source

Question 23

[page] Fine-tuning a model with the Trainer API: [page section] Evaluation: [q] Idiom to get the predictions from the output of trainer.predict()?

Answer 23

import numpy as np
predictions = trainer.predict(tokenized_datasets["”validation””])
preds = np.argmax(predictions.predictions, axis=-1)

Source

Question 24

[page] Fine-tuning a model with the Trainer API: [page section] Evaluation: [q] Example compute_metrics() function on mrpc to be passed to a Trainer? What is the eval_preds argument?

Answer 24

“def compute_metrics(eval_preds):
metric = evaluate.load(““glue””, ““mrpc””)
logits, labels = eval_preds
predictions = np.argmax(logits, axis=-1)
return metric.compute(predictions=predictions, references=labels)
#eval_preds = EvalPredict object = namedtuple with a predictions field and a label_ids field”

Source

Question 25

[q] When not using the Trainer class, write a basic training loop.

Answer 25

"from tqdm.auto import tqdm
progress_bar = tqdm(range(num_training_steps))
model.train()
for epoch in range(num_epochs):
    for batch in train_dataloader:
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()

        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        progress_bar.update(1)"

Source

Question 26

[q] Write a basic evaluation loop using a glue metric.

Answer 26

"import evaluate
metric = evaluate.load(""glue"", ""mrpc"")
model.eval()
for batch in eval_dataloader:
    batch = {k: v.to(device) for k, v in batch.items()}
    with torch.no_grad():
        outputs = model(**batch)
    logits = outputs.logits
    predictions = torch.argmax(logits, dim=-1)
    metric.add_batch(predictions=predictions, references=batch[""labels""])
metric.compute()"

Source