emopcmd.py

usage: emopcmd.py [-h] [-c CONFIG_PATH]
                  {configprint,query,submit,run,upload,testrun,transfer} ...

Options:

-c=/home/docs/checkouts/readthedocs.org/user_builds/emop-controller/checkouts/latest/config.ini, --config=/home/docs/checkouts/readthedocs.org/user_builds/emop-controller/checkouts/latest/config.ini
	path to config file

Sub-commands:

configprint

Undocumented

usage: emopcmd.py configprint [-h] --section SECTION --item ITEM

Options:

--section	config section
--item	config item

query

Undocumented

usage: emopcmd.py query [-h] [--filter FILTER] [--pending-pages]
                        [--avg-runtimes]

Options:

--filter={}	filter to apply
--pending-pages=False
	query number of pending pages
--avg-runtimes=False
	query average runtimes of completed jobs

submit

Undocumented

usage: emopcmd.py submit [-h] [--filter FILTER]
                         [--pages-per-job PAGES_PER_JOB] [--num-jobs NUM_JOBS]
                         [--sim] [--no-schedule] [--no-transfer]

Options:

--filter={}	filter to apply
--pages-per-job
	number of pages per job
--num-jobs	number jobs to submit
--sim=False	simulate job submission
--no-schedule=True
	disable submitting to scheduler
--no-transfer=True
	disable initiating transfer

run

Undocumented

usage: emopcmd.py run [-h] --proc-id PROC_ID [--force-run]

Options:

--proc-id	job proc-id
--force-run=False
	Force run even if output exists

upload

Undocumented

usage: emopcmd.py upload [-h]
                         (--proc-id PROC_ID | --upload-file UPLOAD_FILE | --upload-dir UPLOAD_DIR)

Options:

--proc-id	job proc-id
--upload-file	path to payload file to upload
--upload-dir	path to payload directory to upload

testrun

Undocumented

usage: emopcmd.py testrun [-h] [--filter FILTER]
                          [--num-pages TESTRUN_NUM_PAGES] [--no-upload]

Options:

--filter={}	filter to apply
--num-pages=1	number pages to reserve and run
--no-upload=False
	disable uploading of results

transfer

Undocumented

usage: emopcmd.py transfer [-h] {status,out,in,test} ...

Sub-commands:

status

Undocumented

usage: emopcmd.py transfer status [-h] [--wait WAIT] [--task-id TASK_ID]

Options:

--wait=0	number of seconds to wait for process to finish
--task-id	task ID to query

out

Undocumented

usage: emopcmd.py transfer out [-h] [--proc-id PROC_ID]

Options:

--proc-id

job proc-id

in

Undocumented

usage: emopcmd.py transfer in [-h] [--proc-id PROC_ID] [--filter FILTER]
                              [--wait WAIT]

Options:

--proc-id	job proc-id
--filter={}	filter to apply
--wait=0	number of seconds to wait for process to finish

test

Undocumented

usage: emopcmd.py transfer test [-h] [--wait WAIT]

Options:

--wait=0

number of seconds to wait for process to finish

emopcmd.configprint(args, parser)

emopcmd.main()

emopcmd.query(args, parser)

query command

The query subcommmand behavior is determined by the –pending-pages and –avg-runtimes arguments.

When –pending-pages is passed the number of pending pages is printed. This query can be modified via the –filters argument.

When –avg-runtimes is passed the log files from emop-controller are parsed and average runtimes are printed.

emopcmd.run(args, parser)

run command

The run command is intended to be executed on a compute node. This command performs the actual work of OCRing pages based on the supplied –proc-id value. By default the run command will not run if it detects the PROC_ID has already run, but this can be modified with –force-run. This is useful when a batch job has been requeued.

emopcmd.setup_parser()

emopcmd.submit(args, parser)

submit command

This command will submit jobs based on the various arguments.

Arguments used:

–num-jobs: Number of jobs to submit. The default value is determined based on optimization logic. –pages-per-job: Number of pages per job. The default value if determined based on optimization logic. –no-schedule: Currently unused –sim: Simulate and print the job optimization and commands to use but do not actually submit jobs –filter: Filter to use when querying pending pages

The default logic of this function is to determine optimal number of jobs and pages per job. Once this is determined the necessary number of pages are reserved via Dashboard API. The returned API results are saved to a PROC_ID input file. All the reserved job’s PROC_IDs are then processed and a Globus transfer is initiated and a transfer job is submitted to wait for the transfer to complete. All the reserved page PROC_IDs are submitted as batch jobs and will depend on the transfer job to complete before they start.

emopcmd.testrun(args, parser)

TESTRUN

Reserve pages, run pages and optionally upload pages

emopcmd.transfer_in(args, parser)

transfer in command

The transfer in command will transfer files from a remote Globus endpoint to the cluster Globus endpoint.

When used with –proc-id, the files defined in the PROC_ID file will be transferred.

When used with –filter, the files returned by API query will be transferred.

emopcmd.transfer_out(args, parser)

transfer out command

The transfer out command will transfer files from a cluster Globus endpoint to a remote Globus endpoint.

Currently this command will only transfer out when passed the –proc-id argument, and will parse the PROC_ID file for files to transfer.

emopcmd.transfer_status(args, parser)

transfer status command

When no arguments are given this command will simply verify that both cluster and remote Globus endpoints are activated and usable.

When the –task-id argument is passed this command will query the status of a Globus transfer task. The –wait argument can be used to force this command to wait for the Globus transfer status to be SUCCEEDED or FAILED.

emopcmd.transfer_test(args, parser)

transfer test command

The transfer test command can be used to verify transfers are functional between the cluster and remote Globus endpoints.

The tasks performed: * Perform ls on /~/ on cluster endpoint * Perform ls on /~/ on remote endpoint * Transfer a test file from cluster endpoint to remote endpoint * Display task and wait 2 minutes for task to complete

emopcmd.upload(args, parser)

upload command

The upload command will send data to the Dashboard API.

The –proc-id argument will upload contents of PROC_ID file The –upload-file argument will upload the specified file The –upload-dir argument will upload all JSON files directly under the specified directory.