13. Maintaining ML Solutions

Question 1

What are the steps in ML?

Answer 1

Data:
Extraction (from sources)
Analysis (EDA)
Preparation (transform and feature engineering)
Model:
Training (get the best model)
Evaluation (assess the model quality)
Validation (meet a predefined performance metrics)
Deployment (online & batch):
Serving (RESTful endpoint)
Monitor (Detect anomalies, drift & skew)

Hints:
Data: Elephants Are Playful
Model: Tigers Enjoy Vegetation During Sunny Mornings

Question 2

What are the three levels of MLOps?

Answer 2

Level 0: Manual Phase
Level 1: Strategic automation phase
Level 2: CI/CD automation, transformational phase

Question 3

What are the key features of Level 0?

Answer 3

Manual
ML and MLOps are different teams
No CI/CD/CT
No deploying an entire ML system

Question 3

What are the key features of Level 1?

Answer 3

Orchestrated experimentation
CT
Experiment-operational symmetry
Modular components
CD
Pipeline deployment

Question 4

What are the considerations for triggering retraining?

Answer 4

Training costs
Training time
Delayed training
Scheduled training

Question 4

What are the key features of Level 2?

Answer 4

Pipeline
CI/CD

Question 5

What are the triggers for retraining?

Answer 5

Absolute threshold
Rate of degradation

Question 6

What are the problems for not having a centralised feature store?

Answer 6

Non-reusable: Features created not shared
Governance: Features created by different sources not governed
Cross-collaboration: Features not being shared continue to go separately.
Training and serving differences: Differences may exist between training and serving data.
Productizing features: Lack of automation in features used in experimentation.

Question 6

What is model versioning for?

Answer 6

Deploy an additional model to the existing model.

Question 7

What are the two key features of Feature Store?

Answer 7

Process large feature sets quickly
Access the features with low latency for real-time and batch predictions.

Question 8

Is Vertex AI Feature Store a managed service and scale dynamically?

Answer 8

Yes

Question 9

What model does Feature Store use to store all the data?

Answer 9

Time-series

Question 10

What is the hierarchy of featurestore?

Answer 10

Featurestore > EntityType > Feature

Question 11

What are the two types of ingestions supported by Feature Store?

Answer 11

Batch and streaming ingestion, e.g., BigQuery to Feature Store.

Question 12

What are the two types of retrieving supported by Feature Store?

Answer 12

Batch and online.

Question 13

What are the best practices to use IAM security?

Answer 13

Least privilege
Actively manage service accounts and service account keys
Enable auditing
Check policy management

Question 13

What service do you use to manage permissions to perform various operations?

Answer 13

Identity and Access Management (IAM)

Question 14

What is the specific uses of IAM in Vertex AI?

Answer 14

Google automatically creates several service accounts for Google Cloud Projects. They may have more permissions than required. Use custom service accounts.

Question 15

What is Access Transparency in Vertex AI?

Answer 15

You need logs to track what content and who is accessing it. They may be legal and compliance requirements.
There are two types of access logs. Cloud Audit logs are logs of users from your organisation and Access Transparency logs are logs of Google personnel.

Question 16

What are the common training errors?

Answer 16

Input data not transformed or encoded
Tensor shape mismatched
Out of memory errors because of instance size

Question 17

What are the common serving errors?

Answer 17

Input data not transformed or encoded
Signature mismatched

Question 18

What are the ways to prevent and reduce training and serving errors?

Answer 18

Compute statistics
Infer schema
Detect anomalies

Question 19

What does Vertex AI provide to debug training for both pre-built and custom containers?

Answer 19

Interactive shell

Question 20

What can you inspect with interactive shell during training?

Answer 20

Run tracing and profiling tools
Analyze GPU utilization
Validate IAM permissions for the container