Bulk uploading pages from database

[Michiel Coghe]

Hello Community, 

We are trying to set up a new ELN/Registry for our company on Confluence. For this we need to bulk upload tens of thousands of pages from our database preferably as fast as possible. We've set up a system to call the REST API and also checked out the Confluence CLI and WebDAV options based on other peoples suggestions. All of these options do work but they are very slow (around 4 seconds to build a page). Is there a way this could be done quicker? For example connecting to the underlying Confluence Database?

EDIT: A quick elaboration - We are trying to upload gene sequences and some accompanying meta data in three different groups. Each group has a specific (simple) template with a table, sequence viewer, links to databases, etc. Our tactic now is to create a HTML file for each of the sequences and push these to Confluence using either the API or the runFromCsv command in the Confluence CLI. The CLI takes 3.66 seconds to create a page. The API goes a little quicker, but using this we have some troubles with Time out errors which doesn't make it that feasible to work with.

Thanks in advance,

Michiel

[Piyush A (STR)]

Hello,

Welcome to the community !! (I'm reading your question as - To create multiple pages)

If that takes 4 seconds, then probably you can make it less than 2 seconds. Can you elaborate,if the pages to upload have a predefined template or dynamic data for upload?

Once I had the same with dynamic data, I create a shell script to create 128+ pages with CSV data and JSON data. But that too took 2.3 seconds to create.

From the database - that isn't possible as they are many underlying tables that need to be updated as well, hence, creating via DATABASE is NO.

I'll be watching this question which is quite interesting ask here.

[Michiel Coghe]

Thanks!

A quick elaboration: We are trying to upload gene sequences and some accompanying meta data in three different groups. Each group has a specific (simple) template with a table, sequence viewer, links to databases, etc. Our tactic now is to create a page in HTML format for each of the sequences and push these to Confluence using either the API or the runFromCsv command in the Confluence CLI. The CLI takes 3.66 seconds to create a page. The API goes a little quicker, but using this we have some troubles with Time out errors which doesn't make it that feasible to work with.