Exam Professional Data Engineer topic 1 question 208 discussion

Since cost optimization and minimal latency are key requirements, option D is likely the best choice to meet all the needs: The key reasons option D works well: Using Pub/Sub to ingest votes provides scalable, reliable transport. Loading into Bigtable and BigQuery provides both: Low latency reads from Bigtable for real-time results. Cost effective storage in BigQuery for longer term analysis. Shutting down Bigtable after voting concludes reduces costs. BigQuery remains available for cost-optimized storage and analysis. So you are correct that option D combines the best of real-time performance for queries using Bigtable, with cost-optimized storage in BigQuery. The only additional consideration may be if 3 minutes of Bigtable usage still incurs higher charges than ingesting directly into BigQuery. But for minimizing latency while optimizing cost, option D is likely the right architectural choice given the requirements.

Modern Capabilities: BigQuery’s advancements make it suitable for both real-time and historical querying. Cost Efficiency: No need to spin up and shut down a Bigtable instance. Simplified Workflow: Real-time and post-event data are stored in the same system, reducing the need to synchronize or transfer data between systems.

D. Write votes to a Pub/Sub topic and load into both Bigtable and BigQuery via a Dataflow pipeline. Query Bigtable for real-time results and BigQuery for later analysis. Shut down the Bigtable instance when voting concludes.

D, i do agree with everything MaxNRG said.

Pub/Sub for sure, and Cloud Functions + BigQuery Streaming seems a good solution. Won't use BigTable as need at least 100GB of data (don't thing a voting system could arrive to that amount of data) and needs to "heat" to work right for >10 minutes... and would be $$$ over C solution

Answer is D: - Google Cloud Pub/Sub can manage the high-volume data ingestion. - Google Cloud Dataflow can efficiently process and route data to both Bigtable and BigQuery. - Bigtable is excellent for handling high-throughput writes and reads, making it suitable for real-time vote tallying. - BigQuery is ideal for exact vote counting and deeper analysis once voting concludes.

Handling High-Volume Data Ingestion: Pub/Sub: Decouples vote collection from processing, ensuring scalability and resilience under high load. Dataflow: Efficiently ingests and processes large data streams, scaling as needed. Real-Time Results with Exactly-Once Processing: Bigtable: Optimized for low-latency, high-throughput reads and writes, ideal for real-time partial results. Exactly-Once Semantics: Dataflow guarantees each vote is processed only once, ensuring accurate counts. Cost Optimization: Temporary Bigtable Instance: Running Bigtable only during voting minimizes costs. BigQuery Storage: Cost-effective for long-term storage and analysis.