RE: Content Importer loading duration

Hi,

We have almost 195000 dita files and maps & images (1 GB of dita files & 1.75 GB of Images). We are using Content Importer to load those files to our DEV environment.

It takes around 9 days to load the whole content. In this cutting edge of technologies and agile world, 9 days are too much.

Seems this is not an acceptable duration. What we can do to speed up this loading process? 

(Please do not suggest to chunk our content into multiple sizes we cannot do as we have a lot of cyclical dependencies.)

Suggestions are most welcome. Thanks in advance.

 

Thanks,

PaulGregory

Parents
  • Do you have access to the database servers? If so, export the content of the production database and import it into the development database. The whole process takes about 30 minutes. I do this periodically when I want to make the development environment a "clone" of the production environment, blowing away what's currently in the development database.

    I've included our process as a reference. But if I've learned only two things about databases, it's these:

    • Never blindly run commands found on the internet
    • Always consult a DBA before doing anything with a database

    Further caveats:

    • Our database is Oracle not Microsoft

    A. ON THE PRODUCTION DATABASE SERVER:

    1. Log into the production database and create a .dmp file.

    expdp isource/<password> DUMPFILE=proddb.dmp DIRECTORY=dmpdir SCHEMAS=isource

    2. Copy the file to the development database server.

    B. ON THE DEVELOPMENT DATABASE SERVER:

    1. Make sure the following files exist:

    • the isrcuser.i file
    • $ORACLE_HOME/rdbms/admin/catdbsyn.sql

    2. From SQLPlus, drop and recreate the isource user.

    sqlplus / as sysdba
    drop user isource cascade;
    @/u01/install/isrcuser.i

    (If the second command returns ORA-01940: cannot drop a user that is currently connected, then run shutdown immediate; followed by startup; and retry the command.)

    3. Grant read/write permissions on dmpdir to isource user.

    grant read, write on directory dmpdir TO isource;
    grant all privileges to isource;                    /* Only do this if you run into permission errors later */

    4. Connect to the isource user, run the catdbsyn.sql file, and exit SQLPlus.

    connect isource/<password>;
    @?/rdbms/admin/catdbsyn.sql
    exit

    5. Import the database.

    impdp isource/<password> DUMPFILE=proddb.dmp DIRECTORY=dmpdir SCHEMAS=isource

    6. Start the database and exit SQLPlus.

    sqlplus / as sysdba
    startup;
    exit;

    7. Restart the listener.

    lsnrctl reload

    8. On the application server:

    • restart BackgroundTasks
    • un-register and reregister the crawler
    • restart the index for search
Reply
  • Do you have access to the database servers? If so, export the content of the production database and import it into the development database. The whole process takes about 30 minutes. I do this periodically when I want to make the development environment a "clone" of the production environment, blowing away what's currently in the development database.

    I've included our process as a reference. But if I've learned only two things about databases, it's these:

    • Never blindly run commands found on the internet
    • Always consult a DBA before doing anything with a database

    Further caveats:

    • Our database is Oracle not Microsoft

    A. ON THE PRODUCTION DATABASE SERVER:

    1. Log into the production database and create a .dmp file.

    expdp isource/<password> DUMPFILE=proddb.dmp DIRECTORY=dmpdir SCHEMAS=isource

    2. Copy the file to the development database server.

    B. ON THE DEVELOPMENT DATABASE SERVER:

    1. Make sure the following files exist:

    • the isrcuser.i file
    • $ORACLE_HOME/rdbms/admin/catdbsyn.sql

    2. From SQLPlus, drop and recreate the isource user.

    sqlplus / as sysdba
    drop user isource cascade;
    @/u01/install/isrcuser.i

    (If the second command returns ORA-01940: cannot drop a user that is currently connected, then run shutdown immediate; followed by startup; and retry the command.)

    3. Grant read/write permissions on dmpdir to isource user.

    grant read, write on directory dmpdir TO isource;
    grant all privileges to isource;                    /* Only do this if you run into permission errors later */

    4. Connect to the isource user, run the catdbsyn.sql file, and exit SQLPlus.

    connect isource/<password>;
    @?/rdbms/admin/catdbsyn.sql
    exit

    5. Import the database.

    impdp isource/<password> DUMPFILE=proddb.dmp DIRECTORY=dmpdir SCHEMAS=isource

    6. Start the database and exit SQLPlus.

    sqlplus / as sysdba
    startup;
    exit;

    7. Restart the listener.

    lsnrctl reload

    8. On the application server:

    • restart BackgroundTasks
    • un-register and reregister the crawler
    • restart the index for search
Children
No Data