ACG Notes Flashcards

Question

Describe regional configuration of Cloud SQL

Answer 1

o Regional Replication:  3 read-write replicas  Every mutation requires a write quorum This is different from traditional HA in that it's a read AND a write replica in each zone.

Answer 2

o Design a performant schema o Spread reads/writes around the database, avoid write hot spots o Co-locate compute workloads in the same region o Provision nodes to keep average CPU utilization under 65%

Answer 3

 5 9s SLA – 99.9999%  Reduce latency with distributed data  External consistency • Concurrency control for transactions, this guarantees transactions are executed sequentially even across the globe

Answer 4

 Design a performant schema to avoid hotspots  Co-locate write-heavy compute workloads in the same region as the leader  Spread critical workloads cross two regions  Provision nodes to keep average CPU under 45%

Answer 5

• Data model: o Relational database tables o Strongly typed (you must conform to a strict schema) o Parent-child relationships, declared with primary keys and create an interleaved table

Answer 6

``` • Transactions: o Locking read-write o Read-Only o Partitioned DML o Regular transactions using ANSI SQL best practices ```

Answer 7

o Fully managed Redis instance o 2 tiers:  Basic tier – make sure your app can withstand full data flush  Standard tier – adds cross zone replication and automatic failover

Answer 8

o No need to provision VMs o Scale instances with minimal impact o Private IPs and IAM o Automatic replication and failover

Answer 9

o Session cache – store logins or shopping carts o Message queue – loosely couple micro services o Pub/sub – message queue, but also look at pub/sub

Answer 10

Low latency (use Cloud Bigtable) - - petabyte scale - - single-key rows - - Time series or IoT data Warehouse (use BigQuery) - Petabyte scale - Analytics warehouse - SQL queries

Answer 11

Horizontal scaling (Cloud Spanner) - ANSI SQL - Global replication - High Availability and consistency Vertical Scaling (use Cloud SQL) - MySQL or PostgreSQL - Managed service - High availability

Answer 12

NoSQL - Fully managed document database - Strong consistency - Mobile SDKs and offline data Key/Value - Managed Redis instances - Does what Redis does

Answer 13

distributed implementation of the map and reduce programming model, common interface to program these operations while abstracting away all of the systems management

Answer 14

 Hadoop Common – base files  Hadoop Distributed File System (HDFS) – distributed fault tolerant file system  Hadoop YARN – resource management / job scheduling  Hadoop MapReduce – their own implementation

Answer 15

language for analyzing large datasets, essentially an abstraction for MapReduce o High level framework for running MapReduce jobs on Hadoop clusters

Answer 16

general purpose cluster-computing framework o Pretty much replaced Hadoop, uses Hadoop, much faster  Hadoop stores data in blocks on disk before, during, and after computation  Spark stores data in memory, enabling parallel operations on that data

Answer 17

Hadoop - Slow disk storage - High latency - Used for: slow, reliable batch processing Spark - Fast memory storage - Low latency - Stream processing - 100x faster in-memory - 10x faster on disk

Answer 18

distributed streaming platform, designed for high-throughput and low-latency pub/sub stream of records o Handles >800 billion messages per day at LinkedIn

Answer 19

Kafka - Guaranteed message ordering - Tuneable message retention - Polling (Pull) subscriptions only - Unmanaged Pub/Sub - No message ordering guarantee - 7 day maximum message retention - Pull or Push subscriptions - Managed

Answer 20

o Global messaging and event ingestion | o Serverless and fully managed, processes up to 500 million messages per second

Answer 21

o Multiple pub/sub patterns, one to many, many to one, and many to many o At least once delivery is guaranteed o Can process messages in real-time or batch with exponential backoff o Integrates with Cloud Dataflow

Answer 22

``` o Distributing workloads o Asynchronous workflows – order processing, order, packaging, shipping o Distributing Event Notifications o Distributed Logging o Device Data Streaming ```

Answer 23

``` o Pull (default) – ad-hoc request, messages must be acknowledged, or they will remain at the top of the queue and you won’t get the next message o Push – will send new messages to an endpoint, must be HTTS with a valid cert ```

Answer 24

o Fully supported by Cloud Dataflow o Client libraries for popular languages (python) o Cloud Functions can be triggered by events o Cloud Run to be the receiver of a push sub o IoT Core

Answer 25

o You may receive a message more than once in a single subscription o Message Retention Duration (default 7 days) – undelivered messages are deleted

Answer 26

if No pulls / no pushes – subs expire after 31 days

Answer 27

o Acknowledged messages are no longer available to subscribers o Every message must be processed by a subscription

Answer 28

o Seek – you can rewind the clock and retrieve old messages up to the retention window, you can also use this to seek to a point in the future  Useful in case of an outage most commonly

Answer 29

o Snapshot – save the current state of the queue, this enables replay  Useful if you have new code and you’re not sure what it’ll do, you can snapshot back to a certain point and move forward, then seek back to replay

Answer 30

o Ordering messages – Use timestamps when final order matters, order still isn’t guaranteed but you do have a record of that time.  If you absolutely must guarantee order, consider an alternative system

Answer 31

o Use service accts for authorization, granting per-topic or per-subscription permissions  Grant limited access to publish or consume messages

Answer 32

• Cloud Dataflow – fully managed, serverless ETL tool, using Apache Beam. o Supports:  SQL, Java, and Python  Real-time and batch processing

Answer 33

``` • You can run pipelines on your local machine, this is the preferred way to fix bugs • Pipeline design considerations: o Location of data o Input data structure and format o Transformation objectives o Output data structure and location ```

Answer 34

• ParDo – defines the distributed operation/transformation to be performed on the PCollection of data. These can be user defined functions or pre-defined.

Answer 35

o Data types – may be of any data type but must all be of the same type. The SDK includes built-in encoding o Access – individual access to elements is not supported, transforms are performed on all o Immutable – cannot be changed once created o Boundedness – no limit to the number of elements that a PCollection can contain o Timestamp – associated with every element of the collection, assigned by the creation

Answer 36

o ParDo – generic parallel processing transforms o GroupByKey – processing collections of KVP’s o CoGroupByKey – used when combining multiple key collections, performs relational join o Combine – requires you to provide a function to provide the logic, multiple pre-built functions are available, sum, min, max… o Flatten – Multiple collections become one o Partition – how the elements of the PCollection are split up

Answer 37

o Only users with permission can submit pipelines o Any temp data during execution is encrypted o Any communication between workers happens on a private network o Access to telemetry or metrics is controlled by project permissions

Answer 38

Cloud Dataflow service uses the Dataflow Service Account  Account is automatically created on flow creation  Manipulates job resources on your behalf  Assumes the “Cloud Dataflow service agent role”  Read/write access to project resources (recommended not to change this) Worker instances will use the Controller service account  Used for metadata operations (ex: determine size of file on storage)  Also, able to use user-managed controller service acct, enabling fine grained access-control

Answer 39

• Regional Endpoints – specifying a regional endpoint means all the worker instances will persist in that region, this is best for: o Security and compliance o Data locality o Resiliency

Answer 40

o Used for migrating MapReduce jobs to Cloud Dataflow

Answer 41

o Develop and run Cloud Dataflow jobs from the BigQuery web UI

Answer 42

o Integrates with Apache Beam SQL  Apache Beam SQL: • Can query bounded and unbounded PCollections • Query is converted to a SQL transformation  Cloud Dataflow SQL benefits: • Join streams with BigQuery tables • Query streams or static datasets • Write output to BigQuery for analysis and visualization

Answer 43

• Go with the flow – ideal solution for customers using Apache Beam o Batch == Dataproc & spark o Streaming == Beam and Dataflow

Answer 44

o The pipeline represents the complete set of stages required to read, transform, and write data using the Apache Beam SDK o PCollection – represents a multi-element dataset that is processed by the pipeline

Answer 45

o ParDo – Core parallel processing function of Apache beam which can transform elements of an input PCollection into an output PCollection, can invoke UDF’s o DoFn – template you use to create UDF that are referenced by a ParDo

Answer 46

allows streaming data to be grouped into finite collections according to time or session-based windows  Useful when you need to impose ordering or constraints on pub/sub data

Answer 47

o Watermark – indicates when a Dataflow expects all data in a window to have arrived  Data that arrives with a timestamp that’s inside the window but past the watermark is considered late, policies can decide what happens to this.

Answer 48

o Dataflow is normally the preferred option for data pipelines o Composer may sometimes be used for ad-hoc orchestration or to provide manual control of Dataflow pipelines themselves

Answer 49

• Triggers -- determine when to emit aggregated results as data arrives. o For bounded data, results are emitted after all the input has been processed. o For unbounded data, results are emitted when the watermark passes the end of the window, indicating that the system believes all input data for that window has been processed.

Answer 50

Peta-byte scale, serverless, highly-scalable cloud enterprise DW

Answer 51

o Highly available – automatic data replication o Supports standard SQL – ansi compliant o Supports Federated Data – connects to several external sources o Automatic Backups – automatically replicates data and keeps a 7-day history of changes o Support for Governance and Security – fine-grained IAM o Separation of storage and compute – ACID compliant, stateless compute

Answer 52

Project  Dataset – container for tables/views, think of this like a database • Native table – standard table, where data is held in BQ storage • External tables – backed by storage outside of BQ • Views or virtual tables – created by a SQL Query

Answer 53

Real-time events  Generally streamed using pub/sub, then use cloud dataflow to process these and push to BQ Batch sources  Push files to cloud storage, have cloud dataflow pick up the data and put in BQ

Answer 54

action that is run in BigQuery on your behalf asynchronously | o 4 types: Load / Export / Query / Copy

Answer 55

• Importing data: supported formats: csv, json, Avro, Parquet, ORC, Datastore/Firestore exports

Answer 56

o Control access to data o Reduce query complexity o Can be used to construct logical tables o Enables authorized views – users can have access to different subsets of rows

Answer 57

 Cannot export data from a view  Cannot use JSON API to retrieve data from a view  No UDF’s  Limited to 1k authorized views per dataset

Answer 58

Supports BigTable, Cloud Storage, and Google Drive o Use cases:  Load and clean data in one pass  Small, frequently changing data joined with other tables

Answer 59

``` Limitations:  No guarantee of consistency  Lower query performance  Cannot run export jobs on external data  Cannot query Parquet or ORC formats  Results not cached  Limited to 4 concurrent queries ```

Answer 60

 Ingestion time partition tables |  Partitioned tables

Answer 61

* Partitioned by load or arrival date * Data automatically loaded into date-based partitions (daily) * Tables include the pseudo-column _PARTITIONTIME * use _PARTITIONTIME in queries to limit partitions scanned (in the where)

Answer 62

• Partitioning is based on specific TIMESTAMP or DATE column • Data partitioned based on value supplied in partitioning column • 2 additional partitions: o __null__ o __UNPARTITIONED__ • Use partitioning column in queries

Answer 63

 Improve query performance, less data is read/processed |  Cost control, you pay for all the data processed by a query

Answer 64

o Like creating an index on the table o Supported for both types of partitioning, unsupported for non-partitioned tables o Create a cluster key on frequently accessed columns (order is important, it’s the exact order you’ll be accessing the data)

Answer 65

 Specify clustering columns only on table create, cannot be changed  Clustering columns must be top-level, non-repeated columns  Only supported for partitioned tables  Can specify one to four clustering columns

Answer 66

 Filter clustered columns in the order they were specified:  Avoid using clustered columns in complex filter expressions  Avoid comparing cluster columns to other columns

Answer 67

Slots – unit of computational capacity required to execute SQL queries o Play a role in pricing and resource allocation o Determined by:  Query size  Query complexity (amount of info shuffled) o Automatically managed

Answer 68

Controlling Costs Query Performance Optimizing Storage

Answer 69

 Avoid SELECT * (columns are stored separately)  Use preview options to sample data  Price queries before executing them (price is per byte), there’s a calculator  Using LIMIT does not affect costs, the full columns are read and processed  View costs using a dashboard and query audit logs  Partition by date in your queries  Use streaming inserts with caution, costly, go with bulk loads

Answer 70

• Prune partitioned queries (don’t query partitions you don’t need) • De-normalize data whenever possible • Use external data sources appropriately • Avoid wildcard tables o A wildcard table represents a union of all the tables that match the wildcard expression.

Answer 71

* Avoid repeatedly transforming data via SQL queries * Avoid JS UDF’s * Order query operations to maximize performance * Optimize JOIN patterns

Answer 72

* Avoid self-joins * Avoid data skew * Avoid unbalanced joins * Avoid joins that generate more outputs than inputs * Avoid DML statements that update or insert single rows

Answer 73

• Use expiration settings (tables auto deleted after expiration) • Take advantage of long-term storage o Lower monthly charges apply for data stored in tables or in partitions that have not been modified in 90 days

Answer 74

3 Types of roles relating to BigQuery o Primitive – defined at the project level 3 types: Owner, Editor, Viewer o Predefined – Granular access defined at the service level, GCP managed. Recommended to use this over the primitive roles o Custom – user managed

Answer 75

 Fully managed service  Identify and protect sensitive data at scale  De-identifies data using masking, tokenization, date shifting, and more

Answer 76

• Stackdriver -- fancy CloudWatch logs, you can build a dashboard o Supports metrics from various services including BigQuery

Answer 77

o Linear regression o Binary logistic regression o Multi-class logistic regression o K-means clustering (most recent addition)

Answer 78

``` • Datproc – managed cluster service for Spark and Hadoop o Benefits:  Cluster actions complete in 90 seconds  Pay-per-second, minimum 1 minute  Scale up/down or turn off at will ```

Answer 79

 Submit Hadoop/Spark Jobs  (optionally) enable auto scaling to cope with the load  Output to GCP Services (GCS, BigQuery, BigTable)  Monitor with stackdriver – fully integrated logs

Answer 80

o Single node cluster – limited to the capacity of a single VM and cannot auto scale o Standard cluster: Type and size can be customized o High Availability Cluster

Answer 81

 High availability clusters  Not permitted on single node clusters  HDFS – make sure you have enough primary workers to store all data so you don’t have loss when it scales down  Spark structured streaming  Idle clusters – just delete the idle cluster and make a new one for a new job

Answer 82

• Cloud Storage Connector – run dataproc jobs on GCS instead of HDFS o Cheaper than persistent disk and you get all the GCS benefits o Decouple storage from cluster, decommission cluster when finished, no data loss

Answer 83

great choice for migrating Hadoop and Spark into GCP

Answer 84

Ease of scaling, use GCS instead of HDFS, and the connectors to other GCP services (BigQuery / BigTable)

Answer 85

It provides a software framework for distributed storage and processing of big data using the MapReduce programming model.

Answer 86

Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Answer 87

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Answer 88

Apache Hive is a data warehouse software project built on top of Apache Hadoop for providing data query and analysis. Hive gives an SQL-like interface to query data stored in various databases and file systems that integrate with Hadoop.

Answer 89

Tez is an extensible framework for building high performance batch and interactive data processing applications, coordinated by YARN in Apache Hadoop. Tez improves the MapReduce paradigm by dramatically improving its speed, while maintaining MapReduce's ability to scale to petabytes of data.

Answer 90

A MapReduce program is composed of a map procedure, which performs filtering and sorting, and a reduce method, which performs a summary operation.

Answer 91

• BigTable – managed wide-column NoSQL database (kvp), designed for high throughput with low latency (10,000 reads/sec, 6 ms response) o Scalable and HA  Originally developed internally for web indexing  HBase created an open-source implementation of the BigTable, bought my Microsoft, adopted by Apache o BigTable supports the HBase library for java

Answer 92

• Cloud BigTable Tables: (only index you get is on the row key) o Blocks of contaguous rows are sharded into tables  Tables are stored in Google Colossus – all splitting, merging, rebalancing, happens automatically

Answer 93

(large amount of small data) o Marketing & Financial (stock prices, currency exchange rates) o Time Series & IoT

Answer 94

``` o SQL Support OLTP – Cloud SQL o OLAP – BigQuery o NoSQL Documents – Firestone o In-memory KVP – Memorystore o Realtime DB – Firebase ```

Answer 95

o Instances can run up to 4 clusters o Clusters exist in a single zone Production allows up to: • 30 nodes per project • 1000 tables per instance

Answer 96

Individual cells should be no larger than 10 mb (with history) and no row should be larger than 100 mb)

Answer 97

o Expiry policies define garbage collection:  Expire based on age  Expire based on number of versions

Answer 98

think about what sort of questions you may ask about the database. Scans are the most expensive operation (take the longest).

Answer 99

move data that may normally be in a column and combine it with the row key for querying  Never put a timestamp at the start of a row key, this will make it impossible to balance the cluster

Answer 100

 Domain names  Sequential numbers (Really bad idea, writes will always be on the end)  Frequently updated identifiers (repeatedly updating the same row is not as performant as new rows)  Hashed values, doesn’t really help distribution and doesn’t help with even distribution

Answer 101

 Store related entities in adjacent rows, balance reads/writes  Balanced access pattersns enable linear scaling of performance

Answer 102

Use tall and narrow tables • Use new rows instead of versioned cells • Logically separate event data into different tables where possible • Don’t re-invent the wheel, there’s multiple time series schemas available that have already been proven

Answer 103

 Consider field promotion to the key  Salting  Use the key visualizer to find hotspots

Answer 104

o Eventually consistent data model o Used for: Availability and failover Application isolation Global presence

Answer 105

o Stackdriver metrics can be used for programmatic scaling – not a built-in feature Rebalancing tablets take time; performance may not improve for 20 min Adding nodes does not solve a bad schema/hot node

Answer 106

Good - optimized schema and row key design - large datasets - correct row and column sizing Bad - Datasets smaller than 300GB - short-lived data

Answer 107

If migrating from an on-prem environment, look for HBase, also consider when BigTable beats BigQuery due to the nature of the data, timeseries or latency sensitive information

Answer 108

– under resourced clusters, bad schema design, poorly chosen row-keys

Answer 109

wide tables store multiple columns for a given row-key where the query pattern is likely to require all the information about a single entity. Tall tables suit time-series or graph data and often only have a single column

Answer 110

Jupyter notebooks that can interact with GCP services

Answer 111

Free visualization tool for creating Dashboards and reports

Answer 112

Fully managed workflow service, on top of airflow, task organization system intended to create workflows of various complexity. o Written in python, highly extensible o Central management and scheduling tool o Extensive CLI and web UI tool for managing workflows

Answer 113

* Dataflow is specifically for Batch and Stream data using beam * Composer is a task orchestrator with python Organizing dataflow with composer is a common pattern

Answer 114

each environment is an isolated installation of airflow and its component parts o Can have multiple environments per project but each environment is independent You write DAG’s in python for the scheduler to pick up. This is where you define the order and configuration settings for the workflows.

Answer 115

Pre-trained models  No model training / knowledge of ML required Re-useable models  Model training required, minimal knowledge of ML Build your own  Deep knowledge of ML is required, lots of model training

Answer 116

 identifies objects within images  able to perform facial recognition  Can read printed and handwritten text

Answer 117

 Identifies objects/places/actions in videos, streamed or stored

Answer 118

 Translate between more than 100 different languages

Answer 119

 Converts text to human speech, with 180 voices across 30 languages

Answer 120

 Perform sentiment, entity, content classification…

Answer 121

o Cloud AutoML – train your own custom models to solve specific problems Suite of ML products to facilitate training of custom ML models. ```  Vision  Video Intelligence  Natural Language  Natural Translation  Tables ```

Answer 122

train the model using data that is labeled, the model will be able to infer the feature values on the label

Answer 123

the model is used to uncover structure within the dataset itself Example: Uncover personas within customer data, it will take information that’s similar and group it together

Answer 124

``` o Regression – predict a real number, ex: value of a house o Classification – predict the class from a specified set with a probability score o Clustering – group elements into clusters or groups based on how similar they are ```

Answer 125

• Overfitting – common challenge that has to be overcome training ML models o An overfit model is not generalized, it does not fit unknown data very well. It’s too trained to the training dataset

Answer 126

 Increase training data size  Feature selection – include more or reduce the number of features  Early stopping – not too many iterations on the training data  Cross-validation – take the training data and split it into much smaller sets, these are then used to tune a model, known as folds • K-fold cross validation

Answer 127

o Batch Size o Training epochs – number of times that the full set of training data is ran through o Number of hidden layers in a neural network o Regularization type o Regularization rate o Learning rate

Answer 128

 Model hyperparameters relate directly to the model that is selected  Algorithm – hyperparameters relate to the training model

Answer 129

Open-source neutral network library, high-level API for fast experimentation, supported in TensorFlow’s core library

Answer 130

Google’s open source, end-to-end, ML framework

Answer 131

Tensors represent the flow of information in a neural network

Answer 132

Facilitates sharing of AI resources  Hosted repo of plug and play AI components  End-to-end pipelines  Standard algorithms to solve common problems

Answer 133

o Synchronous mode – responses are returned immediately (online processing) o Asynchronous mode – Only returns results once processing is completed (offline)

Answer 134

* Face Detection – suggests emotional analysis * Image property detection – identify image properties (ex: Dominant colors) * Label Detection – Identify and detect objects, locations, activities, animal species, products….

Answer 135

• Dialogflow – Natural language interaction platform o Used in mobile and web application, devices, and bots o Analyses text or audio inputs o Responds using text or speech

Answer 136

- REST and gRPC - Returns a result after all input audio has been processed - Limited to audio of one minute or less

Answer 137

- Rest and gRPC - initiates a long-running operation - Use the operation to poll for results

Answer 138

- gRPC - Audio data is provided within a gRPC bi-directional stream - results produced while audio is being captured

Answer 139

gRPC (gRPC Remote Procedure Calls[2]) is an open source remote procedure call (RPC) system initially developed at Google in 2015 as the next generation of the RPC infrastructure Stubby. provides features such as authentication, bidirectional streaming and flow control, blocking or nonblocking bindings, and cancellation and timeouts. It generates cross-platform client and server bindings for many languages. Most common usage scenarios include connecting services in a microservices style architecture, or connecting mobile device clients to backend services.

ACG Notes Flashcards

(163 cards)