DataHubApply

This source is a helper over CLI so people can use the helper to apply various metadata changes to DataHub via Managed Ingestion

CLI based Ingestion

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
domain_apply One of array, null	List to apply domains to assets Default: None
domain_apply.DomainApplyConfig DomainApplyConfig
domain_apply.DomainApplyConfig.domain_urn string	Default:
domain_apply.DomainApplyConfig.assets array	List of assets to apply domain hierarchically. Currently only containers and datasets are supported
domain_apply.DomainApplyConfig.assets.string string
owner_apply One of array, null	List to apply owners to assets Default: None
owner_apply.OwnerApplyConfig OwnerApplyConfig
owner_apply.OwnerApplyConfig.owner_urn string	Default:
owner_apply.OwnerApplyConfig.assets array	List of assets to apply owner hierarchically. Currently only containers and datasets are supported
owner_apply.OwnerApplyConfig.assets.string string
tag_apply One of array, null	List to apply tags to assets Default: None
tag_apply.TagApplyConfig TagApplyConfig
tag_apply.TagApplyConfig.tag_urn string	Default:
tag_apply.TagApplyConfig.assets array	List of assets to apply tag hierarchically. Currently only containers and datasets are supported
tag_apply.TagApplyConfig.assets.string string
term_apply One of array, null	List to apply terms to assets Default: None
term_apply.TermApplyConfig TermApplyConfig
term_apply.TermApplyConfig.term_urn string	Default:
term_apply.TermApplyConfig.assets array	List of assets to apply term hierarchically. Currently only containers and datasets are supported
term_apply.TermApplyConfig.assets.string string

Schema

The JSONSchema for this configuration is inlined below.

{
  "$defs": {
    "DomainApplyConfig": {
      "additionalProperties": false,
      "properties": {
        "assets": {
          "description": "List of assets to apply domain hierarchically. Currently only containers and datasets are supported",
          "items": {
            "type": "string"
          },
          "title": "Assets",
          "type": "array"
        },
        "domain_urn": {
          "default": "",
          "title": "Domain Urn",
          "type": "string"
        }
      },
      "title": "DomainApplyConfig",
      "type": "object"
    },
    "OwnerApplyConfig": {
      "additionalProperties": false,
      "properties": {
        "assets": {
          "description": "List of assets to apply owner hierarchically. Currently only containers and datasets are supported",
          "items": {
            "type": "string"
          },
          "title": "Assets",
          "type": "array"
        },
        "owner_urn": {
          "default": "",
          "title": "Owner Urn",
          "type": "string"
        }
      },
      "title": "OwnerApplyConfig",
      "type": "object"
    },
    "TagApplyConfig": {
      "additionalProperties": false,
      "properties": {
        "assets": {
          "description": "List of assets to apply tag hierarchically. Currently only containers and datasets are supported",
          "items": {
            "type": "string"
          },
          "title": "Assets",
          "type": "array"
        },
        "tag_urn": {
          "default": "",
          "title": "Tag Urn",
          "type": "string"
        }
      },
      "title": "TagApplyConfig",
      "type": "object"
    },
    "TermApplyConfig": {
      "additionalProperties": false,
      "properties": {
        "assets": {
          "description": "List of assets to apply term hierarchically. Currently only containers and datasets are supported",
          "items": {
            "type": "string"
          },
          "title": "Assets",
          "type": "array"
        },
        "term_urn": {
          "default": "",
          "title": "Term Urn",
          "type": "string"
        }
      },
      "title": "TermApplyConfig",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "domain_apply": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/DomainApplyConfig"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "List to apply domains to assets",
      "title": "Domain Apply"
    },
    "tag_apply": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/TagApplyConfig"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "List to apply tags to assets",
      "title": "Tag Apply"
    },
    "term_apply": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/TermApplyConfig"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "List to apply terms to assets",
      "title": "Term Apply"
    },
    "owner_apply": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/OwnerApplyConfig"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "List to apply owners to assets",
      "title": "Owner Apply"
    }
  },
  "title": "DataHubApplyConfig",
  "type": "object"
}

Code Coordinates

• Class Name: datahub.ingestion.source.apply.datahub_apply.DataHubApplySource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for DataHubApply, feel free to ping us on our Slack.