Skip to main content
Version: Next

Mode

Certified

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default.
Column-level Lineage✅Supported by default.
Descriptions✅Enabled by default.
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Extract Ownership✅Enabled by default.
Platform Instance✅Enabled by default.
Table-Level Lineage✅Supported by default.

This plugin extracts Charts, Reports, and associated metadata from a given Mode workspace. This plugin is in beta and has only been tested on PostgreSQL database.

Report​

/api/{account}/reports/{report} endpoint is used to retrieve the following report information.

  • Title and description
  • Last edited by
  • Owner
  • Link to the Report in Mode for exploration
  • Associated charts within the report

Chart​

/api/{workspace}/reports/{report}/queries/{query}/charts' endpoint is used to retrieve the following information.

  • Title and description
  • Last edited by
  • Owner
  • Link to the chart in Metabase
  • Datasource and lineage information from Report queries.

The following properties for a chart are ingested in DataHub.

Chart Information​

NameDescription
FiltersFilters applied to the chart
MetricsFields or columns used for aggregation
XFields used in X-axis
X2Fields used in second X-axis
YFields used in Y-axis
Y2Fields used in second Y-axis

Table Information​

NameDescription
ColumnsColumn names in a table
FiltersFilters applied to the table

Pivot Table Information​

NameDescription
ColumnsColumn names in a table
FiltersFilters applied to the table
MetricsFields or columns used for aggregation
RowsRow names in a table

Authentication​

See Mode's Authentication documentation on how to generate an API token and password.

Mode does not support true "service accounts", so you must use a user account for authentication. Depending on your requirements, you may want to create a dedicated user account for usage with DataHub ingestion.

Permissions​

DataHub ingestion requires the user to have the following permissions:

  • Have at least the "Member" role.

  • For each Connection, have at least"View" access.

    To check Connection permissions, navigate to "Workspace Settings" → "Manage Connections". For each connection in the list, click on the connection → "Permissions". If the default workspace access is "View" or "Query", you're all set for that connection. If it's "Restricted", you'll need to individually grant your ingestion user View access.

  • For each Space, have at least "View" access.

    To check Collection permissions, navigate to the "My Collections" page as an Admin user. For each collection with Workspace Access set to "Restricted" access, the ingestion user must be manually granted the "Viewer" access in the "Manage Access" dialog. Collections with "All Members can View/Edit" do not need to be manually granted access.

Note that if the ingestion user has "Admin" access, then it will automatically have "View" access for all connections and collections.

CLI based Ingestion​

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: mode
  config:
    # Coordinates
    connect_uri: http://app.mode.com

    # Credentials
    token: token
    password: pass

    # Options
    workspace: "datahub"
    default_schema: "public"
    owner_username_instead_of_email: False
    api_options:
      retry_backoff_multiplier: 2
      max_retry_interval: 10
      max_attempts: 5

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
password ✅
string(password)
When creating workspace API key this is the 'Secret'.
token ✅
string
When creating workspace API key this is the 'Key ID'.
workspace ✅
string
The Mode workspace username. If you navigate to Workspace Settings > Details, the url will be https://app.mode.com/organizations/<workspace-username>. This is distinct from the workspace's display name, and should be all lowercase.
connect_uri
string
Mode host URL.
Default: https://app.mode.com
exclude_restricted
boolean
Exclude restricted collections
Default: False
ingest_embed_url
boolean
Whether to Ingest embed URL for Reports
Default: True
owner_username_instead_of_email
One of boolean, null
Use username for owner URN instead of Email
Default: True
platform_instance_map
One of string, null
A holder for platform -> platform_instance mappings to generate correct dataset urns
Default: None
tag_measures_and_dimensions
One of boolean, null
Tag measures and dimensions in the schema
Default: True
env
string
The environment that all assets produced by this connector belong to
Default: PROD
api_options
ModeAPIConfig
api_options.max_attempts
integer
Maximum number of attempts to retry before failing
Default: 5
api_options.max_retry_interval
One of integer, number
Maximum interval to wait when retrying
Default: 10
api_options.retry_backoff_multiplier
One of integer, number
Multiplier for exponential backoff when waiting to retry
Default: 2
api_options.timeout
integer
Timout setting, how long to wait for the Mode rest api to send data before giving up
Default: 40
space_pattern
AllowDenyPattern
A class to store allow deny regexes
space_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Code Coordinates​

  • Class Name: datahub.ingestion.source.mode.ModeSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Mode, feel free to ping us on our Slack.