Converting SQLite Data to PostgreSQL - Exportizer Documentation

Here you can find the detailed instruction on how to export/convert SQLite table(s) to PostgreSQL database in Exportizer software.

Below, it is shown how to export data from GUI or command line.

Export Conditions

In most cases, to export from SQLite to PostgreSQL, the following conditions are required:

SQLite3.dll must be accessible.
Exportizer Pro or Exportizer Enterprise must be used.
When using Exportizer Pro, the source and target databases must be connected via ODBC (using ADO interface), so make sure both SQLite ODBC driver and PostgreSQL ODBC driver were installed and the corresponding ODBC data sources were created. The drivers must be both either 32-bit or 64-bit.
When using Exportizer Enterprise, both databases can be connected using FD interface, and there are no ODBC drivers required. Of course, you can use other connection methods available for these types of databases, but note that using FD on the target side can significantly speed up the data exporting.
All involved components, i.e. Exportizer, ODBC drivers (if you use ODBC on source or target side), OLE DB providers, should have the same architecture: 32-bit or 64-bit.

Notes

If your operating system is 64-bit, you can install both 32-bit and 64-bit versions of the Exportizer software and use them independently.
You can invoke the ODBC Data Source Administrator directly from Exportizer (in Open Data Source dialog) when it was launched in administrator mode.

SQLite to PostgreSQL Exporting Preparation

The following preparations are recommended, but not required. If you want to dive into the process and understand how it works, do these steps. Anyway, you can skip them and proceed to the Export Steps below; in this case, all the preparations will be done on the fly, explicitly or implicitly.

Launch Exportizer.
Register your source SQLite database:
- In Exportizer Pro, use ADO interface. The database must be an ODBC DSN, pointed to your SQLite database file.
- In Exportizer Enterprise, use FD interface. In most cases, specifying a pointer to the SQLite file is enough. Sometimes, specifying the Vendor library parameter may be needed: this must be a main DLL from the folder where SQLite installed, e.g. sqlite3.dll.
Register your target PostgreSQL database. The recommended interface is FD, but other options are also available. Basically, you specify a server and a port. Other important options are database, vendor library, and charset. Note: You can register the target database from the Export dialog during the exporting.

Export Steps (GUI)

Preparing Source Data

Open the registered SQLite database or just drug and drop the source SQLite file onto the Exportizer table list.
Choose a dataset or datasets to export:
1. select a table from the table list;
  or
2. open an SQL window, then write and execute an SQL query;
  or
3. select multiple tables in the table list; before doing that, click Select Tables button ;
  or
4. prepare a mix of multiple tables and/or SQL queries: open multiple tables one by one and multiple SQL windows; in SQL windows, write your database queries and execute them to get the result sets.
Please note that exporting multiple datasets at a time is available in Exportizer Enterprise only.

Choosing Target Database and Export Options

If your case is a, b, or c, proceed by clicking Export button . Otherwise, choose Export | Export Open Datasets... menu.
Switch to the Database tab. If this tab is not visible, click Favorite Export Formats button in the top-right corner of the window and make sure the corresponding format is selected.
Select your registered PostgreSQL database as a target database. If you did not register it yet, do it now by clicking the '...' button to the right.
Specify export parameters:
- Turn on the Memory saving mode option and its related options. In case of exporting problems, if you suspect that they can be caused by these options, try different combinations of them.
- Specify the Table name, i.e. name of the target table. For multi-table exporting, you can leave it empty, otherwise, all source datasets will be exported to one destination table; anyways, the source-to-target table mappings are available at the next step.
- Optionally, specify the target table description (comment). For multi-table exporting, leave it empty; you will be able to specify the target descriptions later on the Table Mappings step.
- Choose the Export mode. You can read detailed description for selected export mode to the right (hover mouse over it to see full text). For multi-table exporting, this mode will be applied to the most of the tables, and you can override it for specific tables at the next step. For example, it can be Replace+Insert for the most of tables, and Update or Append+Update for others etc.
- Specify Commit interval. The bigger its value, the faster your exporting process. But too big value may cause memory and other issues. So, try to play with it to find the optimal value for your database before porting the solution to your production environment.

Preparing Source-to-Target Mappings

Click Next to proceed to source-to-target table and/or column mappings or click Export to skip the next steps. When skipping the mappings step, all mappings are built automatically, which sometimes is not desired. When proceeding to the mappings step, initial mappings are built automatically too, but they can be edited when needed. It is recommended to use explicitly built mappings when:

table structure is complex;
new target columns must be added;
preparing a command line for data exporting;
exporting data to already existing tables.

So, when switching to the mappings step, there are two different scenarios depending on how many datasets are involved. If exporting more that one dataset, you work with table mappings, otherwise your work with field/column mappings.

Field Mappings

In field mappings, you create the correspondence between the source and target columns. In addition to source table columns, you can also use calculated fields specified by formulas. It is possible to define full specifications for target columns here, i.e. rename target columns, choose their types, specify NOT NULL constraints, default values, primary key flags, computation formulas, descriptions (comments) etc. These specifications will be applied when the target table needs to be created or overwritten, i.e. when the corresponding export mode is used.

For PostgreSQL identity or computed columns, the source part of the field mappings should be empty.

Editing column mappings for PostgreSQL database

Table Mappings

In table mappings (for multi-dataset exporting), you create the correspondence between the source datasets (tables and/or queries) and target tables. Here, you can specify target table names, their descriptions (comments), and nested field mappings for each table pair.

Logs and Error Handling

Click Next to proceed to export logs and error handling options or click Export to start the export process immediately.

Resolving Export Performance Issues

In the case of the slow performance, there are some actions you can try:

First of all, make sure the target database is connected via FD interface. Using FD on the target side lets you utilize batch SQL commands execution which can speed up the exporting process drastically.
Try different combinations of Memory saving mode options and Commit interval setting. It is recommended to turn on the Memory saving mode, Use SQL parameters and Batch mode options and set the value of Commit interval from several hundreds to several thousands depending on you target server resources.
Finally, you might want to read all the data exporting recommendations.

Exporting SQLite to PostgreSQL from Command Line

Here are some examples of exporting data from an SQLite file to PostgreSQL database.

Exporting an SQLite Table to PostgreSQL Database

"C:\Program Files (x86)\Vitaliy Levchenko\Exportizer Enterprise 10\exptizer.exe" /export /ExportType=DATABASE /ExportMode=REPLACE+INSERT /IncludeMemo /MemorySaving /UseSQLParameters /UseBatchMode /SrcDBInterface=FD /SrcDBKind=FILE /SrcDBDriver=SQLite /SrcDB=C:\TEST\employees.db /SrcTableName=BONUSES /TrgDBInterface=FD /TrgDBKind=DSN /TrgDBDriver=POSTGRESQL "/TrgServer=222.333.2.14" /TrgPort=5432 /TrgDB=dwh /TrgDBUserName=dwh_master /TrgDBPassword=dwh_master_password /TrgTableName=bonuses

SQLite to PostgreSQL Exporting in Silent Mode

Let's improve the above example by running it in silent mode, when no windows are shown. Also, let's add a log file to control exporting process:

"C:\Program Files (x86)\Vitaliy Levchenko\Exportizer Enterprise 10\exptizer.exe" /silent /export /ExportType=DATABASE /ExportMode=REPLACE+INSERT /IncludeMemo /MemorySaving /UseSQLParameters /UseBatchMode /SrcDBInterface=FD /SrcDBKind=FILE /SrcDBDriver=SQLite /SrcDB=C:\TEST\employees.db /SrcTableName=BONUSES /TrgDBInterface=FD /TrgDBKind=DSN /TrgDBDriver=POSTGRESQL "/TrgServer=222.333.2.14" /TrgPort=5432 /TrgDB=dwh /TrgDBUserName=dwh_master /TrgDBPassword=dwh_master_password /TrgTableName=bonuses /LogFile=C:\Test\export.log

Exporting All Tables from SQLite File

Exporting all tables from SQLite file to PostgreSQL database in silent mode:

"C:\Program Files (x86)\Vitaliy Levchenko\Exportizer Enterprise 10\exptizer.exe" /silent /export /ExportType=DATABASE /ExportMode=REPLACE+INSERT /IncludeMemo /MemorySaving /UseSQLParameters /UseBatchMode /SrcDBInterface=FD /SrcDBKind=FILE /SrcDBDriver=SQLite /SrcDB=C:\TEST\employees.db /SrcTableName=* /TrgDBInterface=FD /TrgDBKind=DSN /TrgDBDriver=POSTGRESQL "/TrgServer=222.333.2.14" /TrgPort=5432 /TrgDB=dwh /TrgDBUserName=dwh_master /TrgDBPassword=dwh_master_password /TrgTableName=* /LogFile=C:\Test\export.log

Notes

These are just basic command lines, and you can improve them by adding other command line parameters in accordance with the documentation. For example, consider explicit using of the field or table mappings by adding the following parameters:

/FieldMappingsFile Specify the file containing field mappings, i.e. the correspondence between the source and target fields/columns, when exporting one dataset.
/TableMappingsFile Specify the file containing table mappings, i.e. the correspondence between the source and target tables, when exporting multiple tables. The table mappings may optionally contain nested field mappings for certain (or all) tables.

Alternative Scenario Using SQL

If you have some reason not to export data to PostgreSQL directly, you can export the SQLite data in Exportizer to a SQL script first, and then load data from the script in Exportizer or using an external SQL tool. And, of course, you can export data to SQL script either by GUI or from the command line.

But please note, that when choosing exporting via SQL script, your export process will have two phases: first, exporting the data to SQL script; second, loading data from the script to your target database. So, this way can be less effective, especially for large datasets or when automating the data exporting. Another disadvantage of this way is that BLOB data will not be transferred.

Anyway, you can try both export scenarios and select the fastest and/or the most convenient one.

See also