01. Import and Export using List Services

List Services Is a data store that can be used to store a high volume of list-based contact records with any number and type of columns. It can associate events with every record and provide millisecond access to them. Once the records are imported into the store the following features are available:

• Search by record’s columns, events, geolocation, full text and predefined filter

• Analytics and dashboards on records and associated events

• High-performance purl generation

• Sftp Import and export of data with optional PGP encryption

• Scheduled export to download, using SFTP and HTTP protocols

• Access to index tuning for better performance

• Secure access by user, partner and javascript SDK

• Highly available across multiple availability regions and cloud providers

• Optional sharding for accounts over 10M

• Integrated by studio workflow to send outbound (e.g., email, sms etc) to selected records

How to access List Services in the MindFire Portal

You can open the menu description by clicking on the three dots at the bottom of the menu, as shown in the screenshot below. The List Services option is marked in the image. Once you know which one it is, you should be able to click on the List Services option without opening the entire menu. Please note that in order to access List Services, you will need to be assigned the correct permission by your power user (usually this is the administrator for your account).

What are the different parts of the List Services?

When you log into the List Services for the first time, you will be able to see the options below. 

Dashboards - This section is for reporting and dashboards that show the performance of the lists in terms of responses.

Imports - In this section, you will import the lists that you will use for your multi-channel communications.

Indexes - In this section, you will define the fields that you use for searching the data uploaded to List Services.

Search and Analyze - In this section, you can search and retrieve data previously uploaded to List Services, using one or more of the search criteria defined in the Indexes section.

Settings - Set up various settings associated with List Services.

Once you have uploaded data into List Services, you will also be able to see three more options - Exports, Deletes and Remote Access.

How to set up the first-time import to a List Services account

While it is easy to import data to List Services, you will need to create some setup data associated with it during the first-time import into an account.

From the main menu of the portal, please select List Services, click on Imports and then click on the New Import button as shown above. That will open up the following dialog box. Click on the blue File to Upload button and select the upload file from your local drive. The file can be comma or tab deleted, and it can be a zip file or a PGP encrypted file in addition to being a CSV file.

Once you have selected the file to upload, the system will default the filename to the import name. The user can always change the import name. The user can also click on the File Preview option to review the file before uploading it, as shown in the screenshot below.

At this time you should be able to visit the Settings, the PURL Columns and the DEDUP Columns tabs and complete the import setup. These are one-time settings for each new account that is being used with List Services, and we will not be required to do them again when we upload data to this same account again. The various one-time settings options are explained below.

The following options are available in the Settings tab.

Generate PURL: this should be turned on for most MindFire campaigns. In a situation where you do not wish to use PURLs or personalized QR codes for a specific account, you can turn it off. Please note that this is a setting that cannot be changed for this account. We recommend turning this ON.

Same PURL: This switch indicates whether a specific contact, as identified by the dedupe  columns (to be described later in this section) is uploaded as part of different lists, will have the same personalized URL or not. For example, let us consider that the account number is the dedupe column. List A has a contact John Doe with account number 12345 and is uploaded today. A month later, the same contact with the same account number is uploaded again. If the “Same PURL” switch is turned off, John Doe will have a separate PURL each time the record is uploaded. This is particularly useful when each list has separate offers and both offers are valid simultaneously. If the “Same PURL” switch is on, then there will be only one PURL for John Doe, irrespective of the list, and only the most recent offer will be available. We do not have any specific recommendations here - it depends on the use case. If the contacts are being recirculated in multiple lists over a period of time, it is better to turn this OFF.

Cast Type: Since the NO SQL database has no pre-defined schema, this option is particularly useful to generate field types associated with the database schema based on the data supplied, e.g. numeric, string, date etc. We recommend turning this ON.

Persist Event: This is to persist events in the List Services database for reporting and dashboard. We recommend turning this ON.

Reject on Error: This option decides whether a file will be fully rejected on error or get partially loaded. The errors can be viewed in the Search and Analyze section and then can be corrected and re-uploaded. Our recommendation for this option is to keep it ON, especially if the “Same PURL” option is OFF unless the user has a good process in place to separate the erroneous records, edit them and re-upload.

Auto Export: This option allows the user to define the criteria for adding fields in the automated export following the import process. Once the auto-export is set up, it really streamlines the import process for future lists. Some very useful examples of fields to be added during the auto-export process are the complete URL’s using PURLs and domains, QR code strings, UTM parameters etc. The following section gets into the details of the Auto Export process. We recommend turning it ON, especially if you will be sending the exported file for printing.

How to Setup Auto-Export in List Services

First of all, the auto-export for an account is a one-time activity and needs to be done only during the initial import to an account. For all subsequent accounts, the user can edit it if any information has changed, e.g. using a different domain for a QR code - however, no need to touch it if all the information is still the same.

When the user turns on the Auto Export switch, a section of the dialog box shows up asking the user to provide the Export Definition. The user should click on the three dots marked in the screenshot below with the red arrow to bring up the subsequent export configuration screen.

In the auto-export definition, click the three dots in the top right corner of the field definitions, as shown in the screenshot below.

This opens up the following dialog. All CSV fields + PURLs is the most natural option to select. The IDs are necessary if the data is being fed to an external application such as the Wallet Pass system which will use the IDs to record external events.

Once you select one of the two options, you will see a screen similar to the one below, with all the CSV fields and the PURL field showing the generated PURLs for each record. You can delete any of the fields if you do not want them as part of the export. For example, in the example in the screenshot below, as shown with the red arrows, the fields titled Text1 can be deleted if we do not want them in the export file.

At this time you can add more fields in the export file if necessary. If you click on the field name of the last available row, it will show you a list of system fields that you can add, and a custom option, as shown in the screenshot below. We already discussed the field called _id. Let us discuss the custom option now.

The custom option allows you to build additional fields by using the values from existing fields. In the example of the screenshot below, we are using the custom option to build a QR code URL string that will be used to print the QR code on a postcard. The fields are specified using {fieldname} construct, and you can also use fixed values, e.g. list name, mail date etc.

Once you click on SUBMIT, you will see the following screen with the field definitions. At this time you can optionally add a notification email by clicking on the three dots at the top right corner of the Notifications box. At the bottom left in the drop-down highlighted in red, you can select comma, tab or pipe delimited output. You can also choose to zip the output file if you like. Finally, click on SAVE.

After this, you need to specify the PURL columns, as in the screenshot below. The columns must be valid columns that are present in the header brown of the CSV file being uploaded.

Next - set up the dedupe column.

You can also set up SFTP and PGP encryption and make this a completely hands-free automated process. However, SFTP and PGP will be covered in a different article.

Once you fill in the PURL and Dedupe fields, you should be able to click on the SUBMIT button and the import process will get started. Once the import is complete, you will get a notification email if you set one up. In any case, you should be able to see the imported file and its status in the Import section of the List Services, as shown in the screenshot below.

You should now go to the Export section and you will see the export file, as generated, with the additional fields that you defined in the field definition section, as shown in the screenshot below. You can click on the download button to the right, as shown below with the red arrow, and download the exported file to your local drive.

Once you download the file, you will see a file with the same name as the Export name (which is inherited from the import name, which is the same as the original file name) in your Download folder. At this point, you have completed the first-time import and auto-export of a mail file using List Services, to a new account.

Moving forward, if you are not changing the auto export configuration, each time you should just have to select the file to be uploaded from your local drive and process it. You can change the auto-export configuration if needed, e.g., the domain changes in the QR code string.