Feast

Important Capabilities

Capability	Status	Notes
Descriptions	✅	Enabled by default.
Detect Deleted Entities	✅	Enabled by default via stateful ingestion.
Schema Metadata	✅	Enabled by default.
Table-Level Lineage	✅	Enabled by default.

This plugin extracts:

• Entities as MLPrimaryKey
• Fields as MLFeature
• Feature views and on-demand feature views as MLFeatureTable
• Batch and stream source details as Dataset
• Column types associated with each entity and feature

CLI based Ingestion

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: feast
  config:
    # Coordinates
    path: "/path/to/repository/"
    # Options
    environment: "PROD"

sink:
  # sink configs

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
path ✅ string	Path to Feast repository
enable_owner_extraction boolean	If this is disabled, then we NEVER try to map owners. If this is enabled, then owner_mappings is REQUIRED to extract ownership. Default: False
enable_tag_extraction boolean	If this is disabled, then we NEVER try to extract tags. Default: False
environment string	Environment to use when constructing URNs Default: PROD
fs_yaml_file One of string(path), null	Path to the feature_store.yaml file used to configure the feature store Default: None
owner_mappings One of array, null	Mapping of owner names to owner types Default: None
owner_mappings.map map(str,string)
stateful_ingestion One of StatefulIngestionConfig, null	Stateful Ingestion Config Default: None
stateful_ingestion.enabled boolean	Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False

Schema

The JSONSchema for this configuration is inlined below.

{
  "$defs": {
    "StatefulIngestionConfig": {
      "additionalProperties": false,
      "description": "Basic Stateful Ingestion Specific Configuration for any source.",
      "properties": {
        "enabled": {
          "default": false,
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "title": "Enabled",
          "type": "boolean"
        }
      },
      "title": "StatefulIngestionConfig",
      "type": "object"
    }
  },
  "properties": {
    "stateful_ingestion": {
      "anyOf": [
        {
          "$ref": "#/$defs/StatefulIngestionConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Stateful Ingestion Config"
    },
    "path": {
      "description": "Path to Feast repository",
      "title": "Path",
      "type": "string"
    },
    "fs_yaml_file": {
      "anyOf": [
        {
          "format": "path",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Path to the `feature_store.yaml` file used to configure the feature store",
      "title": "Fs Yaml File"
    },
    "environment": {
      "default": "PROD",
      "description": "Environment to use when constructing URNs",
      "title": "Environment",
      "type": "string"
    },
    "owner_mappings": {
      "anyOf": [
        {
          "items": {
            "additionalProperties": {
              "type": "string"
            },
            "type": "object"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Mapping of owner names to owner types",
      "title": "Owner Mappings"
    },
    "enable_owner_extraction": {
      "default": false,
      "description": "If this is disabled, then we NEVER try to map owners. If this is enabled, then owner_mappings is REQUIRED to extract ownership.",
      "title": "Enable Owner Extraction",
      "type": "boolean"
    },
    "enable_tag_extraction": {
      "default": false,
      "description": "If this is disabled, then we NEVER try to extract tags.",
      "title": "Enable Tag Extraction",
      "type": "boolean"
    }
  },
  "required": [
    "path"
  ],
  "title": "FeastRepositorySourceConfig",
  "type": "object"
}

Code Coordinates

• Class Name: datahub.ingestion.source.feast.FeastRepositorySource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Feast, feel free to ping us on our Slack.