Scripted Loading

Table of Contents

  1. Loadfile

In many cases it is necessary to load a file from the server rather that using the user interface to define a feed and to schedule it. monsterDB provides 2 integration points for this:

  • The Java API incorporating a bulk loader,
  • Load a file directly from the command prompt.

This document discusses the monsterDB command prompt loader.

Loadfile

LoadFile is currently available via the Command Line (ie windows CMD or *nix bash) only:

java -jar monsterDB.jar -h host -p port -u username -pw password -d aDatabase -c aCollection -f file -r recordType -o options

Where aCollection is a collection (domain context) that can have existing documents within it.

RecordType is the custodian tablename, if you omit this, the record will not be matchable.

Filename/url a local (to the server you are on!) file that can be read by the current user, or a URL such as http://… https://… or ftp:.//… pointing at a file or zip file on the local network or accessible on the internet.

options are a quoted text string containing a json object, property values for this are:

  • encoding – file character encoding, defaults to UTF8
  • rootNode – in XML this will be used to extract the documents, every XML element that is named with this will be converted to a document.
  • extractPattern – when a zip file is found by the URL then this pattern will . be used to extract the files needed from within it.
  • quote – The character used to quote fields of data that might contain the delimiter.
  • delimiter -delimiter character(s) between fields in the rows of data in a delimited file – ie ,
  • fileType – ie CSV, XML, XLS which indicates the type of the file that will eventually be read.
  • sheetno – with an XLS sheet this is the 0 based sheet number to read.

Plus if the file is a URL starting http, https or ftp then the following options can be used

  • httpusername – username used to login to the remote http server
  • httppassword – associated http password
  • httpheaders – any headers required to create a http connection
  • ftpusername – ftp username to login to the remote server
  • ftppassword – ftp password

The following is an example to load a local XML file:

java -jar monsterDB.jar -h host -p port -u username -pw password -d aDatabase -c fuzzyCollection -f leismall.xml -r GLEIF -o "{fileType: "XML", rootNode: "LEIRecord"}"

And to load a XML file that has been ZIPPED and placed on a website:

java -jar monsterDB.jar -d newdb -c fuzzy2 -t GLEIF -f https://leidata.gleif.org/api/v1/concatenated-files/lei2/20190805/zip -o "{"fileType": "zip", "rootNode": "LEIRecord", "extractPattern": ".*.xml"}"