This article is for Flatfile's Portal 2.0, Portal 3.0, and Workspaces. If you'd like to check out the latest version of the Flatfile Data Exchange Platform, click here!

If your users are regularly importing large data files, there are a few recommendations we have to ensure the process is as smooth as possible. These are not all necessary, especially for smaller files, but can be helpful in speeding up imports of larger files.

Flatfile can handle files up to 1GB, but as the file gets larger it will take more time to process. CSV file formats can be processed much faster than Excel files, so we recommend importing in the CSV file format when possible for the best experience, especially with large files.

Check that your users are able to import over a stable internet connection.

When building out templates against which users will be importing larger files, be mindful of how you are building out your data hooks. FieldHooks will be most effective here in processing your data quickly. If possible, avoid external API calls as these will increase the amount of time it takes Data Hooks® to run.

Remove columns that will not be used in the import from the file before importing.

When getting your data from a large import from Flatfile, if you aren't using our onData method there are two different API methods we recommend over the GET Download an Upload endpoint that will be more helpful for large files.

To get the completed rows from the import, we recommend the GET Records endpoint. The default row count that is returned here is 50, but this can be set much higher, for example 50,000 rows at once, to retrieve your data quickly.

If you would rather download a file to egress your processed data, you can export using the initializeSheetExportGraphQL mutation. This mutation will take your sheet ID , which can be retrieved from a Webhook response, and validationState (which would be "approved", "review", or "dismissed") and provide an exportId, which can then be used in a following REST call to fetch the CSV using/export/<exportId>.