T10 - Practical Flashcards

Question 1

Q

read csv using pandas

Answer

A

Let’s break down your code step by step:

```python
import pandas as pd
~~~

✅ This imports the pandas library, which is essential for working with structured data like CSV files. It is commonly imported using the alias pd.

```python
df = pd.read_csv(‘veteran.csv’)
~~~

✅ This line:

Reads the contents of the file veteran.csv.
Assumes the file is in the current working directory.
Loads it into a pandas DataFrame named df, which is like a table with rows and columns.

If the file path or name is incorrect, this line will raise a FileNotFoundError.

```python
df.info()
~~~

✅ This gives a summary of the DataFrame. Here’s what it typically shows:

🔍 Example Output

```python
<class ‘pandas.core.frame.DataFrame’>
RangeIndex: 100 entries, 0 to 99
Data columns (total 5 columns):
# Column Non-Null Count Dtype
— —— ————– —–
0 ID 100 non-null int64
1 Age 100 non-null float64
2 Gender 98 non-null object
3 Disease 100 non-null object
4 Survived 100 non-null int64
dtypes: float64(1), int64(2), object(2)
memory usage: 4.0 KB
~~~

This tells you:

The structure of your dataset.
Which columns might have missing values (important for cleaning).
What data types each column is (important for analysis or modeling).

Let me know if you want to see the first few rows of the data using df.head(), or need help cleaning or analyzing it.

Question 2

Q

change data type:

df['DemCluster'] = df['DemCluster'].astype(str)
   
type(df['DemCluster'][0])

Answer

A

Let’s go through your two lines of code and explain what’s happening:

🧩 Line 1:

```python
df[‘DemCluster’] = df[‘DemCluster’].astype(str)
~~~

✅ This converts the entire DemCluster column in the DataFrame to string (text) data type.

Why do this? Sometimes a column contains numbers like 1, 2, 3 — but you want to treat them as categories, not numerical values. Converting them to str ensures they are no longer treated as numeric.
Example: 1 becomes '1'

🧩 Line 2:

```python
type(df[‘DemCluster’][0])
~~~

✅ This checks the data type of the first element in the DemCluster column (at index 0).

df['DemCluster'][0] fetches the first value in the column.
type(...) returns the Python data type of that value.

Since you converted the entire column to str in the first line, the output will be:

```python
<class ‘str’>
~~~

✅ Summary

Let me know if you want to convert it back to numeric, or check for specific values!

Question 3

Q

change DemHomeOwner into binary 0/1 variable

dem_home_owner_map = {'U':0, 'H': 1}
df['DemHomeOwner'] = df['DemHomeOwner'].map(dem_home_owner_map)

Answer

A

Certainly! Let’s break this code down step by step:

🧩 Code:

```python
# change DemHomeOwner into binary 0/1 variable
dem_home_owner_map = {‘U’: 0, ‘H’: 1}
df[‘DemHomeOwner’] = df[‘DemHomeOwner’].map(dem_home_owner_map)
~~~

✅ What’s Happening:

🔸 dem_home_owner_map = {'U': 0, 'H': 1}

This defines a dictionary that maps the original categorical values:
- 'U' → 0 (probably means Unknown or Not homeowner)
- 'H' → 1 (probably means Homeowner)

🔸 df['DemHomeOwner'].map(dem_home_owner_map)

This replaces each value in the DemHomeOwner column using the dictionary:
- Wherever there’s 'U', it becomes 0
- Wherever there’s 'H', it becomes 1

🔸 df['DemHomeOwner'] = ...

This updates the DemHomeOwner column in the DataFrame with the newly mapped binary values.

🧠 Why Do This?

Many machine learning algorithms and statistical models require numerical input, not strings.
Binary encoding like this helps convert categorical variables into numeric format so they can be used in models.

💡 Example Before and After

Before:

After:

| DemHomeOwner |
| ———— |
| 1 |
| 0 |
| 1 |

Let me know if you want to handle more than two categories or missing values too!

DemHomeOwner |
| ———— |
| H |
| U |
| H |

Question 4

Q

df['DemHomeOwner'].value_counts()

Answer

A

The line:

```python
df[‘DemHomeOwner’].value_counts()
~~~

is used to count the frequency of each unique value in the DemHomeOwner column.

🧠 What It Does:

Returns a summary table showing how many times each value appears.
Useful for checking:
- Class distribution (especially after mapping)
- Data imbalance
- Whether the mapping (e.g., 'U' → 0, 'H' → 1) worked correctly

📊 Example Output (after mapping):

```python
1 150
0 50
Name: DemHomeOwner, dtype: int64
~~~

This means:

150 rows have the value 1 (homeowners)
50 rows have the value 0 (non-homeowners or unknown)

💡 Tip:

If you’re checking before mapping, and the column still has strings:

```python
df[‘DemHomeOwner’].value_counts()
~~~

might output:

```python
H 150
U 50
~~~

Let me know if you want to visualize this with a bar chart or check for missing values too!

Question 5

Q

return the count of a specific value in a variable, e.g., zero 0

df['DemMedIncome'].value_counts().get(0)

Question 6

Q

print(df['DemMedIncome'].value_counts(dropna=False))

Answer

A

0 2357
43100 10
48772 8
72339 8
46250 8
…
55640 1
54626 1
80329 1
36031 1
59813 1
Name: DemMedIncome, Length: 4463, dtype: int64

Question 7

Q

Imputing Missing Values

Question 8

Q

# DemAge has 7279 non-NaN records, while the total number of records in this data set is 9686 
print('non-NaN count = ', df['DemAge'].count())

impute missing values in DemAge with its mean
df.fillna({'DemAge': df['DemAge'].mean()}, inplace=True)

check the number of values again
print('After filling out the missing values, non-NaN count = ', df['DemAge'].count())

Answer

A

Let’s go through your code step by step to understand what’s happening:

🧩 Step 1: Check how many non-missing values exist

```python
print(‘non-NaN count = ‘, df[‘DemAge’].count())
~~~

✅ This line counts the number of non-NaN (non-missing) values in the DemAge column.

df['DemAge'].count() only counts values that are not NaN.
Suppose it prints:
```
python
non-NaN count = 7279
```
That means there are 9686 total rows, and 2407 values are missing (9686 - 7279).

🧩 Step 2: Fill missing values with the mean

```python
df.fillna({‘DemAge’: df[‘DemAge’].mean()}, inplace=True)
~~~

✅ This line replaces all missing values (NaNs) in the DemAge column with the mean of the column.

df['DemAge'].mean() calculates the average of all existing (non-NaN) values.
fillna(...) fills the missing values with that mean.
inplace=True means the changes are saved directly in the df DataFrame without needing to assign it back.

🧩 Step 3: Check again

```python
print(‘After filling out the missing values, non-NaN count = ‘, df[‘DemAge’].count())
~~~

✅ Now, count() should return the total number of rows, because all missing values in DemAge have been filled.

It should now print:

python
After filling out the missing values, non-NaN count = 9686

✅ Summary

This is a simple and common data imputation technique used to prepare data for modeling or analysis.

Let me know if you’d like to try a different method (like median or interpolation) instead of mean!

Question 9

Q

# for gender, before one hot encoding. .head() is used to display first 10 records.
print(df['DemGender'].head(10))

Answer

A

0 F
1 F
2 M
3 M
4 M
5 M
6 M
7 U
8 F
9 U
Name: DemGender, dtype: object

Question 10

Q

# after one hot encoding
demo_df = pd.get_dummies(df['DemGender'])
demo_df.head(10)

Answer

A

Let’s walk through your code:

🧩 Code:

```python
demo_df = pd.get_dummies(df[‘DemGender’])
demo_df.head(10)
~~~

✅ What it does:

pd.get_dummies(df['DemGender'])

Performs one-hot encoding on the DemGender column.
It converts categorical text values (like 'Male', 'Female', 'Other', etc.) into binary columns.

✅ For example, if DemGender contains:

Male
Female
Female
Male
Other

After get_dummies(), you get:

Each original category becomes a new column, and rows are marked with 1 if they belong to that category.

✅ demo_df.head(10)

This just shows the first 10 rows of the newly created one-hot encoded DataFrame.

📌 Why use one-hot encoding?

Machine learning models typically cannot work with text, so categorical variables must be converted to numeric format.

Optional Tip:

To join this back with the main DataFrame:

```python
df = pd.concat([df, demo_df], axis=1)
~~~

Let me know if you want to drop the original DemGender column afterward or use this in a model!

Female | Male | Other |
| —— | —- | —– |
| 0 | 1 | 0 |
| 1 | 0 | 0 |
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 0 | 0 | 1 |

Question 11

Q

# one hot encoding all categorical variables
# all numerical variables are automatically excluded
# number of columns after the conversion should increase significantly
print("Before:", len(df.columns))

one hot encoding
df = pd.get_dummies(df)
print("After:", len(df.columns))

Answer

A

Great! Let’s break down exactly what’s happening in your code:

🧩 Code:

```python
print(‘Before:’, len(df.columns))

df = pd.get_dummies(df)

print(“After:”, len(df.columns))
~~~

✅ What This Does:

🔹 print('Before:', len(df.columns))

Prints the number of columns in the DataFrame before applying get_dummies().

🔹 df = pd.get_dummies(df)

Performs one-hot encoding on all categorical columns in the DataFrame df.
Any column that has text or categorical values (e.g., 'Gender', 'MaritalStatus') will be converted into separate binary (0/1) columns — one per unique value.

For example:

After get_dummies(df) becomes:

| Gender_Female | Gender_Male | MaritalStatus_Married | MaritalStatus_Single |
| ————– | ———— | ———————- | ——————— |
| 0 | 1 | 0 | 1 |
| 1 | 0 | 1 | 0 |
| 1 | 0 | 0 | 1 |

🔹 print("After:", len(df.columns))

Prints the number of columns after the transformation — it should increase, especially if there were multiple categorical columns with many unique values.

🧠 Why Use This?

One-hot encoding is necessary because:

Most machine learning algorithms don’t accept text — they require numerical input.
It helps represent categorical features numerically, without introducing order or bias (unlike label encoding).

⚠️ Important Note:

If you apply pd.get_dummies() to the entire DataFrame, it will transform all categorical columns, and leave numeric columns as they are.
Be cautious: this might explode your column count if some categorical columns have many unique values.

Let me know if you want to:

Keep only certain columns for one-hot encoding
Avoid the “dummy variable trap” (multicollinearity)
Or just inspect the new column names!

Question 12

Q

# show all columns information
print("Before dropping two columns, #columns = ", len(df.columns))

Answer

A

This line:

```python
print(“Before dropping two columns, #columns = “, len(df.columns))
~~~

is used to print the total number of columns in your DataFrame df before you drop (remove) two columns.

🧠 Why it’s useful:

When you’re cleaning or transforming data — especially when doing something like:

```python
df.drop([‘col1’, ‘col2’], axis=1, inplace=True)
~~~

—you want to verify that:

The number of columns decreases appropriately
You didn’t drop more or fewer columns than intended
Your DataFrame structure remains correct

🔍 Example:

Suppose before dropping:

```python
df.columns = [‘Age’, ‘Gender’, ‘Income’, ‘City’, ‘OwnsCar’]
~~~

Then this line will print:

Before dropping two columns, #columns =  5

After dropping:

```python
df.drop([‘City’, ‘OwnsCar’], axis=1, inplace=True)
print(“After dropping two columns, #columns = “, len(df.columns))
~~~

You’ll get:

After dropping two columns, #columns =  3

Let me know if you want to list the actual column names before and after dropping, or track changes to column types!

Question 13

Q

```

df.drop([‘ID’,’TargetD’], a

Answer

A

Let’s break down this line of code:

🧩 Code:

```python
df.drop([‘ID’, ‘TargetD’], axis=1, inplace=True)
~~~

✅ What it does:

🔹 df.drop([...], axis=1)

The .drop() method removes columns or rows from a DataFrame.
axis=1 tells pandas to drop columns (not rows).
You’re telling pandas to drop two specific columns: 'ID' and 'TargetD'.

🔹 inplace=True

This means the change is made directly to the original DataFrame (df).
Without inplace=True, you’d need to assign the result back to df, like:
```
python
df = df.drop([...], axis=1)
```

💡 Why drop these columns?

'ID': Often a unique identifier. It doesn’t carry useful information for modeling or analysis (since it’s just a label).
'TargetD': Sounds like a target variable, possibly unused in the current analysis — e.g., if you’re working with a different target like 'TargetB'.

🧠 Tip: Confirm the drop

After dropping the columns, you can confirm with:

```python
print(df.columns)
~~~

or:

```python
print(“After dropping columns, total columns = “, len(df.columns))
~~~

Let me know if you’d like help inspecting what’s left in your DataFrame or deciding which other columns may be unneeded.

Question 14

Q

When to drop variables

Question 15

Q

from sklearn import preprocessing

Returns a numpy array for input X
x_vals = X.values 

Scaling each feature to the default range (0, 1)
min_max_scaler = preprocessing.MinMaxScaler() 
x_scaled = min_max_scaler.fit_transform(x_vals)

Store the scaled data in a DataFrame 
X2 = pd.DataFrame(x_scaled)
print(X2)

Answer

A

Let’s walk through your code step by step — you’re scaling your data using Min-Max normalization with sklearn.

🧩 Code Explanation:

```python
from sklearn import preprocessing
~~~

Imports the preprocessing tools from scikit-learn, including MinMaxScaler.

```python
x_vals = X.values
~~~

Converts your DataFrame X (which likely contains only numeric feature columns) into a NumPy array.
This is required because MinMaxScaler works on arrays, not DataFrames.

```python
min_max_scaler = preprocessing.MinMaxScaler()
~~~

Creates an instance of the Min-Max scaler.
This scaler will transform features by scaling them to a given range (default is 0 to 1).

```python
x_scaled = min_max_scaler.fit_transform(x_vals)
~~~

Fits the scaler to your data (computes min and max for each feature).
Then transforms the data so each value is scaled according to:

$$
x_{\text{scaled}} = \frac{x - \text{min}(x)}{\text{max}(x) - \text{min}(x)}
$$

This results in a new NumPy array x_scaled where every column’s values are between 0 and 1.

```python
X2 = pd.DataFrame(x_scaled)
print(X2)
~~~

Converts the scaled NumPy array back into a pandas DataFrame.
You print it to see the normalized values.

🔍 Why scale data?

Many machine learning algorithms (like KNN, SVM, neural networks) perform better or faster when features are scaled.
Min-max scaling ensures that no feature dominates because of its range (e.g., salary vs. age).

🧠 Tip: Preserve column names

By default, your new DataFrame X2 won’t have the original column names. To keep them:

```python
X2 = pd.DataFrame(x_scaled, columns=X.columns)
~~~

Let me know if you’d like to use standard scaling instead or need to apply this to a test set as well!

Question 16

Q

Answer

Study These Flashcards

A

T10 - Practical Flashcards

(16 cards)