You architect a system to analyze seismic dat
a. Your extract, transform, and load (ETL) process runs as a series of MapReduce jobs on an Apache Hadoop cluster. The ETL process takes days to process a data set because some steps are computationally expensive. Then you discover that a sensor calibration step has been omitted. How should you change your ETL process to carry out sensor calibration systematically in the future?
You currently use a SQL-based tool to visualize your data stored in BigQuery The data visualizations require the use of outer joins and analytic functions. Visualizations must be based on data that is no less than 4 hours old. Business users are complaining that the visualizations are too slow to generate. You want to improve the performance of the visualization queries while minimizing the maintenance overhead of the data preparation pipeline. What should you do?
To improve the performance of visualization queries while minimizing maintenance overhead, using materialized views is the most effective solution. Here's why option C is the best choice:
Materialized Views:
Materialized views store the results of a query physically, allowing for faster access compared to regular views which execute the query each time it is accessed.
They can be automatically refreshed to reflect changes in the underlying data.
Incremental Updates:
The incremental updates capability of BigQuery materialized views ensures that only the changed data is processed during refresh operations, significantly improving performance and reducing computation costs.
This feature helps maintain up-to-date data in the materialized view with minimal processing time, which is crucial for data that needs to be no less than 4 hours old.
Performance and Maintenance:
By using materialized views, you can pre-compute and store the results of complex queries involving outer joins and analytic functions, resulting in faster query performance for data visualizations.
This approach also reduces the maintenance overhead, as BigQuery handles the incremental updates and refreshes automatically.
Steps to Implement:
Create Materialized Views:
Define materialized views for the visualization queries with the necessary configurations
CREATE MATERIALIZED VIEW project.dataset.view_name
AS
SELECT ...
FROM ...
WHERE ...
Enable Incremental Updates:
Ensure that the materialized views are set up to handle incremental updates automatically.
Update the data visualization tool to reference the materialized views instead of running the original queries directly.
BigQuery Materialized Views
Optimizing Query Performance
You are building new real-time data warehouse for your company and will use Google BigQuery streaming inserts. There is no guarantee that data will only be sent in once but you do have a unique ID for each row of data and an event timestamp. You want to ensure that duplicates are not included while interactively querying dat
a. Which query type should you use?
Which of these statements about BigQuery caching is true?
When query results are retrieved from a cached results table, you are not charged for the query.
BigQuery caches query results for 24 hours, not 48 hours.
Query results are not cached if you specify a destination table.
A query's results are always cached except under certain conditions, such as if you specify a destination table.
You are working on a sensitive project involving private user dat
a. You have set up a project on Google Cloud Platform to house your work internally. An external consultant is going to assist with coding a complex transformation in a Google Cloud Dataflow pipeline for your project. How should you maintain users' privacy?
