Exam Professional Data Engineer topic 1 question 269 discussion

- It utilizes Data Catalog's native support for both BigQuery datasets and Pub/Sub topics. - For PostgreSQL tables running on a Compute Engine instance, you'd use Data Catalog APIs to create custom entries, as Data Catalog does not automatically discover external databases like PostgreSQL.

Data Catalog is the best choice. But for catalogging PostgreSQL it is better to use a connector when available, instead of using API. https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_unsupported_data_sources

This is the correct approach. Data Catalog provides automatic discovery and metadata extraction for both BigQuery datasets and Pub/Sub topics as native Google Cloud services. You'll only need to use the Data Catalog APIs to manually catalog the PostgreSQL tables running on Compute Engine, as this is a non-native data source. This maximizes automation while minimizing development effort.

Why C is Correct? BigQuery datasets → ✅ Automatically cataloged in Data Catalog Pub/Sub topics → ✅ Automatically cataloged in Data Catalog PostgreSQL on Compute Engine → ❌ Not automatically cataloged Requires a custom connector to extract metadata and push it to Data Catalog. Option B (using Data Catalog APIs manually) is not enough because PostgreSQL metadata isn’t natively supported. Why Not B? Option B suggests using Data Catalog APIs manually for PostgreSQL. However, Data Catalog does not natively support PostgreSQL metadata extraction. You need a custom connector to first extract PostgreSQL schema information, then push it to Data Catalog.

https://cloud.google.com/data-catalog/docs/concepts/overview#automatic_cataloging_of_assets https://cloud.google.com/data-catalog/docs/concepts/overview#catalog-non-google-cloud-assets

This section explains it clearly: https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_unsupported_data_sources.

This is C. To clarify some issues below with B, the links provided by supporters of B actually do say that it's preferable to use a community connector where available, and to only use the API when the case is genuinely not supported by community connectors. In this case it's Postgresql, so it's supported, see here for full list: https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_on-premises_data_sources So this would be B if it was something like Q+ or some genuinely unsupported database, but postgres is supported for community connector.

Data Catalog automatically catalogs metadata from Google Cloud sources such as BigQuery, Vertex AI, Pub/Sub, Spanner, Bigtable, and more. To catalog metadata from non-Google Cloud systems in your organization, you can use the following: Community-contributed connectors to multiple popular on-premises data sources Manually build on the Data Catalog APIs for custom entries

raaad mostly correct and we can check his description supporting his answer so we can go with it .Cheers mate

I’m voting for C because the documentation states that Postgres is a custom connector developed by the community.

BigQuery Datasets and Pub/Sub Topics: Google Data Catalog can automatically catalog metadata from BigQuery and Pub/Sub, making it easy to discover and manage these data assets without additional development effort. PostgreSQL Tables: While Data Catalog does not have built-in connectors for PostgreSQL, you can use the Data Catalog APIs to manually catalog the PostgreSQL tables. This requires some custom development but is manageable compared to creating custom connectors for everything.

B. Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use Data Catalog APIs to manually catalog PostgreSQL tables.

Option B leverages Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics, which streamlines the process and reduces manual effort. Using Data Catalog APIs to manually catalog PostgreSQL tables ensures consistency across all data assets while minimizing development and configuration efforts.

I vote for c as per Integrate on-premises data sources To integrate on-premises data sources, you can use the corresponding Python connectors contributed by the community: under the link https://cloud.google.com/data-catalog/docs/integrate-data-sources

In the opction C, the expression "Use custom connectors to manually catalog PostgreSQL tables." is refering to the use case of Google when you want to use "Community-contributed connectors to multiple popular on-premises data sources". As you can see, this connectors are for ON-PREMISSES data sources ONLY. In this case the Postgres is in a VM in the cloud. Thus, the option correct is B.

option B, there's no need to build a custom connector now, postgreSQL is now supported https://github.com/GoogleCloudPlatform/datacatalog-connectors-rdbms/tree/master/google-datacatalog-postgresql-connector

Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use Data Catalog APIs to manually catalog PostgreSQL tables.