Your new customer has requested daily reports that show their net consumption of Google Cloud compute resources and who used the resources. You need to quickly and efficiently generate these daily reports. What should you do?
A.
Do daily exports of Cloud Logging data to BigQuery. Create views filtering by project, log type, resource, and user.
B.
Filter data in Cloud Logging by project, resource, and user; then export the data in CSV format.
C.
Filter data in Cloud Logging by project, log type, resource, and user, then import the data into BigQuery.
D.
Export Cloud Logging data to Cloud Storage in CSV format. Cleanse the data using Dataprep, filtering by project, resource, and user.
A. Do daily exports of Cloud Logging data to BigQuery. Create views filtering by project, log type, resource, and user.
You cannot import custom or filtered billing criteria into BigQuery. There are three types of Cloud Billing data tables with a fixed schema that must further drilled-down via BigQuery views.
Reference:
https://cloud.google.com/billing/docs/how-to/export-data-bigquery#setup
Sadly, it's unclear what:
"Do daily exports of Cloud Logging data to BigQuery"
Does that means creating job? or you would be creating a sink into destination?
For generating daily reports that show net consumption of Google Cloud compute resources and user details, the most efficient approach would be:
A. Do daily exports of Cloud Logging data to BigQuery. Create views filtering by project, log type, resource, and user.
Here's why this option is the most effective:
Integration with BigQuery: BigQuery is a powerful tool for analyzing large datasets. By exporting Cloud Logging data directly to BigQuery, you can leverage its fast querying capabilities and advanced analysis features.
Automated Daily Exports: Setting up automated daily exports to BigQuery streamlines the reporting process, ensuring that data is consistently and efficiently transferred.
Creating Views for Specific Filters: By creating views in BigQuery that filter data by project, log type, resource, and user, you can tailor the reports to the specific needs of your customer. Views also simplify repeated analysis by encapsulating complex SQL queries.
Efficiency and Scalability: This method is highly efficient and scalable, handling large volumes of data without the manual intervention required for CSV exports and data cleansing.
Option B (exporting data in CSV format) and Option D (using Cloud Storage and Dataprep) are less efficient due to the additional steps and manual handling involved. Option C is similar to A but lacks the specificity of creating views directly in BigQuery for filtering, which is a more streamlined approach.
I see A as quite inefficient as you are exporting ALL logs (hundreds of thousands) to bq and the filtering them with views. I would go for C, assuming that it does not involve doing it manually but rather creating a SINK with the correct filters and then using BQ Dataset as sink destination. But a lot of assumptions are taking place here as I believe the questions does not provide much context.
I almost got it wrong by choosing C. By doing C, that means we will manually filter first one by one. We should just import them all and filter using BigQuery
B and D do not consider the log type field.
C looks good and I would go for it.
However, A looks equally good and I've found a CloudSkillsBoost lab that is exactly describing what answer A does, i.e. exporting logs to BQ and then creating a VIEW. https://www.cloudskillsboost.google/focuses/6100?parent=catalog I think the advantage of exporting complete logs (i.e. filtering them after they reach BQ) is that in case we would want to adjust the reporting in the future, we would have the complete logs with all fields available, whereas with C we would need to take extra steps.
You need to quickly and efficiently generate these daily reports by using Materialized view /View
A materialized view is the best solution and having filtered value with a view is good solution so A is an answer
2nd tought: Definitely A. If you go to google documentation for export billing, you see a message that "Exporting to JSON or CSV is obsolet. Use Big Query instead".
Also why A? Look
https://cloud.google.com/billing/docs/how-to/export-data-bigquery
https://cloud.google.com/billing/docs/how-to/bq-examples#total-costs-on-invoice
You can make a fast report template al Data Studio that read a Big Query view.
A comment regarding the links you provided (and not the correctness of the selected answer).
Using Cloud Billing is something different than detecting compute consumption data from Cloud Logging.
In fact, manual exporting to CSV (and JSON) is possible through the Logs Explorer interface (I think without user data break-down):
🔗 https://cloud.google.com/logging/docs/view/logs-explorer-interface#download_logs
The Google Cloud Storage bucket where you would like your reports to be delivered.
You can select any Cloud Storage bucket for which you are an owner, including buckets that are from different projects. This bucket must exist before you can start exporting reports and you must have owner access to the bucket. Google Cloud Storage charges for usage, so you should review the Cloud Storage pricesheet for information on how you might incur charges for the service.
https://cloud.google.com/compute/docs/logging/usage-export
Ans is C
https://cloud.google.com/logging/docs/export/aggregated_sinks
D isn't correct because Cloud storage is used as a sink when logs are in json format not csv. https://cloud.google.com/logging/docs/export/aggregated_sinks#supported-destinations
A voting comment increases the vote count for the chosen answer by one.
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one.
So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
AWSandeep
Highly Voted 2 years, 4 months ago22c1725
Most Recent 1 month, 1 week agoMaxNRG
1 year agoMaxNRG
1 year agoMaxNRG
1 year agoAman47
1 year agoAman47
1 year agovaga1
1 year, 7 months agoSiant_137
1 year, 8 months agomidgoo
1 year, 9 months agomaci_f
1 year, 11 months agozellck
2 years agoAtnafu
2 years, 1 month agoAtnafu
2 years agohauhau
2 years, 1 month agodevaid
2 years, 3 months agodevaid
2 years, 2 months agoNicolasN
2 years, 1 month agoAHUI
2 years, 3 months agoTNT87
2 years, 3 months agojkhong
2 years, 1 month agoTNT87
1 year, 12 months agoTNT87
2 years, 3 months agochangsu
2 years, 3 months agopluiedust
2 years, 3 months agoWasss123
2 years, 3 months ago