Skip to content

Add connections

Spotlight can connect to a range of external data systems at your organization. See Supported data for a list of supported data types, names, and sizes.

Each Spotlight connection is to a particular source of data on one of these systems, for instance to a particular database among the many databases that may be stored on a single server.

You can connect a data system to Spotlight from the home page or as part of adding an individual dataset reference to a Workspace.

Add from home page

Add from home page#

  • Click the 'Add new' button button and select the "Data" option.
  • Click on the icon for the type of system you wish to connect.
  • Enter connection details for the system and any user credentials you may need to access it.
  • Click the 'Connect' button button.

Your data system is now connected to Spotlight!

Add from inside a Workspace

Add from inside a Workspace#

  • Either click the 'Add' button button from the Wrokspace detail page and select the "Data" option or click the 'Add data' button from the Workbench.
  • Click on the icon for the type of system you wish to connect.
  • Enter connection details for the system and any user credentials you may need to access it.
  • Click the 'Connect' button button.
  • Select the individual database table or data file you wish to add as a dataset.
  • Click the 'Add connection' button button to finish creating the dataset.

This will add the new connection to Spotlight and add a reference to the dataset you selected into your Workspace.

Connections are not duplicated in Spotlight. If you add a connection that already exists, you will be redirected to the existing connection detail page and can immediately begin browsing its datasets.

Introspection#

Once a system is connected, Spotlight begins to index datasets on that system and make them available for use in Spotlight (see Concepts of Spotlight: Introspection)

Only table-based systems like databases and data warehouses are introspected. File-based systems (Datameer X, S3, and your local computer when uploading files) are not introspected.

Connecting by source type#

Upload file

Upload file#

Click the 'Upload file' button button to upload data files or documents from your computer directly to Spotlight.

Drag and drop any files you wish to upload or use the add buttons to browse for files and folders on your computer. When a folder is uploaded, each file will be turned into a separate document or dataset depending on its file type. A progress bar will indicate the upload status.

Files in CSV, XLS/XLSX/XLSB, JSON, and Parquet formats become datasets in Spotlight and can be viewed and interacted with inside a Workspace's Workbench. All other formats (including PDF, DOC, and image files) are handled as documents.

See uploading files for full details on handling of different file types, uplolad limitations, and other features of uploading files.

Any files you upload will live in your company's Spotlight environment and all the other Spotlight users in your organization will be able to read and use them in their own work. Please consider the confidentiality of files before uploading them to Spotlight.

Snowflake

Snowflake#

You can build a Spotlight dataset from any of the databases in your Snowflake warehouse. Each table in the database is treated as a separate dataset. Once you add one table as a Spotlight dataset, Spotlight will introspect the rest of the tables in that individual database and make them available as datasets.

For better performance with more complex queries, Spotlight uses a native Snowflake connector to push down aggregations and joins when possible. Note that this connector does not require staging write access permissions.

These screenshots show the information to provide:

Connect to Snowflake Warehouse

After providing the connection information for your Snowflake warehouse you will see one of the two dialog screens shown below based on whether you are adding this connection from the Spotlight home page or from inside a Workspace.

Select Database (when adding from Spotlight home page)

If you are adding this connection from the Spotlight home page, the second dialog screen will simply ask you to select a Database, Schema, and Warehouse.

Select Database and Table information (when adding from Workspace)

If you are adding this connection from inside a Workspace, the second dialog screen will ask for Database, Schema, Warehouse, as well as the particular Table you wish to reference in your Workspace.

Datameer X

Datameer X#

You can build a dataset from any uploaded file, import job, or workbook on your Datameer X system. Each kept workbook sheet is treated as a separate dataset and must have been Run at least once before they can be added to Spotlight. Once you add one sheet as a dataset, Spotlight will introspect the rest of the workbooks in the corresponding folder and make their kept sheets available as datasets.

Note that only worksheets saved in parquet format can be connected to Spotlight. Storing in Parquet has been the default since Datameer X v6.3.

For instructions on configuring your Datameer X system to connect with Spotlight, see Datameer X + Datameer Spotlight.

These screenshots show the information to provide:

Connect to Datameer X

After providing the connection information for your Datameer X instance you will see one of the two dialog screens shown below based on whether you are adding this connection from the Spotlight home page or from inside a Workspace.

Select folder (when adding from the Spotlight home page)

If you are adding this connection from the Spotlight home page, the second dialog screen will simply ask you to select a Folder.

Select and describe File (when adding from a Workspace)

If you are adding this connection from inside a Workspace, the second dialog screen will ask for Folder as well as the particular File you wish to reference in your Workspace.

S3

S3#

You can build a Spotlight dataset from one or more CSV, JSON, or Parquet files in your S3 bucket. Multiple files can be added at once if they share a schema. Spotlight does not introspect S3 connections so each time you add from S3 only a single dataset is created. If multiple files are specified, they will be added as rows to the single new dataset created in Spotlight.

Files can be named anything as long as the file name does not contain the backtick (`) character or begin with either a space or an underline (_) character.

Once you select the file(s) to add to Spotlight, a message will appear at the bottom of your screen to let you know the contents are being copied from S3 and transformed into a Spotlight dataset. You will be notified in your activities panel and with another message at the bottom of your screen once the file has completed transfering to Spotlight.

These screenshots below show the information to provide.

Connect to S3

Enter the Access Key and Secret for your Account or IAM role here. If you are adding from a public bucket, select No Authentication.

Specify Bucket and File

Enter your bucket and file name(s) here. To specify multiple files, add the * wildcard to the folder path and/or file name in the Object Key field.

For example, /my.bucket/extracts/*.csv will match all the CSV files in the extracts directory while /my.bucket/extracts/*/*.json will match all JSON files in all directories under extracts.

Database

Database#

You can build a Spotlight dataset from a table in any of the these database systems: Microsoft SQL Server, MySQL, Oracle, PostgreSQL, Redshift, or Teradata. Each database table is treated as a separate dataset. Once you add one table as a Spotlight dataset, Spotlight will introspect the rest of the tables in that individual database and make them available as datasets.

These screenshots show the information to provide:

Connect to your database server

After providing the connection information for your database, you will see one of the two dialog screens showns below based on whether you are adding this connection from the Spotlight home page or from inside a Workspace.

Select Database (when adding from Spotlight home page)

If you are adding this connection from the Spotlight home page, the second dialog screen will simply ask you to select a Database.

Select and describe table (when adding from a Workspace)

If you are adding this connection from inside a Workspace, the second dialog screen will ask for Database as well as the particular Table you wish to reference in your Workspace.

Tableau Catalog

Tableau Catalog#

You can build a Spotlight dataset from a table in any database listed in your Tableau Catalog. Each database table is treated as a separate dataset. Once you add one table as a Spotlight dataset, Spotlight will introspect the rest of the tables in that individual database and make them available as datasets.

These screenshots show the information to provide:

Connect to Tableau Catalog

Select Database

Connect to Database and select Table

Once you connect to your Tableau Catalog and select which database you wish to use, you will then have to connect to that database in a new dialogue. We will use the Tableau Catalog to fill in the database connection information but you will still need to enter your own user credentials and then select which table to use as usual. For other information on making a database connection, see the "Database" help above.