Management system¶

The management system described below is optional: experiments can be run one by one using artiq_run, and the controllers can run stand-alone (without a controller manager). For their very first steps with ARTIQ or in simple or particular cases, users do not need to deploy the management system.

Components¶

Master¶

The master is responsible for managing the parameter and device databases, the experiment repository, scheduling and running experiments, archiving results, and distributing real-time results.

The master is a headless component, and one or several clients (command-line or GUI) use the network to interact with it.

Controller manager¶

Controller managers (started using the artiq_ctlmgr command that is part of the artiq-comtools package) are responsible for running and stopping controllers on a machine. There is one controller manager per network node that runs controllers.

A controller manager connects to the master and uses the device database to determine what controllers need to be run. Changes in the device database are tracked by the manager and controllers are started and stopped accordingly.

Controller managers use the local network address of the connection to the master to filter the device database and run only those controllers that are allocated to the current node. Hostname resolution is supported.

Warning

With some network setups, the current machine’s hostname without the domain name resolves to a localhost address (127.0.0.1 or even 127.0.1.1). If you wish to use controllers across a network, make sure that the hostname you provide resolves to an IP address visible on the network (e.g. try providing the full hostname including the domain name).

Command-line client¶

The command-line client connects to the master and permits modification and monitoring of the databases, monitoring the experiment schedule and log, and submitting experiments.

Dashboard¶

The dashboard connects to the master and is the main way of interacting with it. The main features of the dashboard are scheduling of experiments, setting of their arguments, examining the schedule, displaying real-time results, and debugging TTL and DDS channels in real time.

Experiment scheduling¶

Basics¶

To use hardware resources more efficiently, potentially compute-intensive pre-computation and analysis phases of other experiments are executed in parallel with the body of the current experiment that accesses the hardware.

Priorities and timed runs¶

When determining what experiment to begin executing next (i.e. entering the preparation stage), the scheduling looks at the following factors, by decreasing order of precedence:

Experiments may be scheduled with a due date. If there is one and it is not reached yet, the experiment is not eligible for preparation.
The integer priority value specified by the user.
The due date itself. The earlier the due date, the earlier the experiment is scheduled.
The run identifier (RID), an integer that is incremented at each experiment submission. This ensures that, all other things being equal, experiments are scheduled in the same order as they are submitted.

Pauses¶

In the run stage, an experiment may yield to the scheduler by calling the pause() method of the scheduler. If there are other experiments with higher priority (e.g. a high-priority timed experiment has reached its due date), they are preemptively executed, and then pause() returns. Otherwise, pause() returns immediately. To check whether pause() would in fact not return immediately, use artiq.master.scheduler.Scheduler.check_pause().

The experiment must place the hardware in a safe state and disconnect from the core device (typically, by calling self.core.comm.close() from the kernel, which is equivalent to artiq.coredevice.core.Core.close()) before calling pause().

Accessing the pause() and check_pause() methods is done through a virtual device called scheduler that is accessible to all experiments. The scheduler virtual device is requested like regular devices using get_device() (self.get_device()) or setattr_device() (self.setattr_device()).

check_pause() can be called (via RPC) from a kernel, but pause() must not.

Multiple pipelines¶

Multiple pipelines can operate in parallel inside the same master. It is the responsibility of the user to ensure that experiments scheduled in one pipeline will never conflict with those of another pipeline over resources (e.g. same devices).

Pipelines are identified by their name, and are automatically created (when an experiment is scheduled with a pipeline name that does not exist) and destroyed (when they run empty).

Git integration¶

The master may use a Git repository for the storage of experiment source code. Using Git has many advantages. For example, each result file (HDF5) contains the commit ID corresponding to the exact source code that produced it, which helps reproducibility.

Even though the master also supports non-bare repositories, it is recommended to use a bare repository so that it can easily support push transactions from clients. Create it with e.g.:

$ mkdir experiments
$ cd experiments
$ git init --bare

You want Git to notify the master every time the repository is pushed to (updated), so that it is rescanned for experiments and e.g. the GUI controls and the experiment list are updated.

Create a file named post-receive in the hooks folder (this folder has been created by the git command), containing the following:

#!/bin/sh
artiq_client scan-repository

Then set the execution permission on it:

$ chmod 755 hooks/post-receive

You may now run the master with the Git support enabled:

$ artiq_master -g -r /path_to/experiments

Push commits containing experiments to the bare repository using e.g. Git over SSH, and the new experiments should automatically appear in the dashboard.

Note

If you plan to run the ARTIQ system entirely on a single machine, you may also consider using a non-bare repository and the post-commit hook to trigger repository scans every time you commit changes (locally). The ARTIQ master never uses the repository’s working directory, but only what is committed. More precisely, when scanning the repository, it fetches the last (atomically) completed commit at that time of repository scan and checks it out in a temporary folder. This commit ID is used by default when subsequently submitting experiments. There is one temporary folder by commit ID currently referenced in the system, so concurrently running experiments from different repository revisions is fully supported by the master.

The dashboard always runs experiments from the repository. The command-line client, by default, runs experiment from the raw filesystem (which is useful for iterating rapidly without creating many disorganized commits). If you want to use the repository instead, simply pass the -R option.

Scheduler API reference¶

The scheduler is exposed to the experiments via a virtual device called scheduler. It can be requested like any regular device, and then the methods below can be called on the returned object.

The scheduler virtual device also contains the attributes rid, pipeline_name, priority and expid that contain the corresponding information about the current run.

class artiq.master.scheduler.Scheduler(ridc, worker_handlers, experiment_db, log_submissions)[source]¶

check_pause(rid)[source]¶

Returns True if there is a condition that could make pause not return immediately (termination requested or higher priority run).

The typical purpose of this function is to check from a kernel whether returning control to the host and pausing would have an effect, in order to avoid the cost of switching kernels in the common case where pause does nothing.

This function does not have side effects, and does not have to be followed by a call to pause.

check_termination(rid)[source]¶: Returns True if termination is requested.

delete(rid)[source]¶: Kills the run with the specified RID.

get_status()[source]¶

Returns a dictionary containing information about the runs currently tracked by the scheduler.

Must not be modified.

request_termination(rid)[source]¶: Requests graceful termination of the run with the specified RID.

submit(pipeline_name, expid, priority=0, due_date=None, flush=False)[source]¶

Submits a new run.

When called through an experiment, the default values of pipeline_name, expid and priority correspond to those of the current run.

Client control broadcasts (CCBs)¶

Client control broadcasts are requests made by experiments for clients to perform some action. Experiments do so by requesting the ccb virtual device and calling its issue method. The first argument of the issue method is the name of the broadcast, and any further positional and keyword arguments are passed to the broadcast.

CCBs are used by experiments to configure applets in the dashboard, for example for plotting purposes.

class artiq.dashboard.applets_ccb.AppletsCCBDock(*args, **kwargs)[source]¶

ccb_create_applet(name, command, group=None, code=None)[source]¶

Requests the creation of a new applet.

An applet is identified by its name and an optional list of groups that represent a path (nested groups). If group is a string, it corresponds to a single group. If group is None or an empty list, it corresponds to the root.

command gives the command line used to run the applet, as if it was started from a shell. The dashboard substitutes variables such as $python that gives the complete file name of the Python interpreter running the dashboard.

If the name already exists (after following any specified groups), the command or code of the existing applet with that name is replaced, and the applet is restarted and shown at its previous position. If not, a new applet entry is created and the applet is shown at any position on the screen.

If the group(s) do not exist, they are created.

If code is not None, it should be a string that contains the full source code of the applet. In this case, command is used to specify (optional) command-line arguments to the applet.

This function is called when a CCB create_applet is issued.

ccb_disable_applet(name, group=None)[source]¶

Disables an applet.

The applet is identified by its name, after following any specified groups.

This function is called when a CCB disable_applet is issued.

ccb_disable_applet_group(group)[source]¶

Disables all the applets in a group.

If the group is nested, group should be a list, with the names of the parents preceding the name of the group to disable.

This function is called when a CCB disable_applet_group is issued.

Front-end tool reference¶

artiq_master¶

ARTIQ master

usage: artiq_master
       [-h]
       [--version]
       [--device-db DEVICE_DB]
       [--dataset-db DATASET_DB]
       [-g]
       [-r REPOSITORY]
       [--experiment-subdir EXPERIMENT_SUBDIR]
       [-v]
       [-q]
       [--log-file LOG_FILE]
       [--log-backup-count LOG_BACKUP_COUNT]
       [--name NAME]
       [--log-submissions LOG_SUBMISSIONS]

Named Arguments¶

--version: print the ARTIQ version number
--name: friendly name, displayed in dashboards to identify master instead of server address
--log-submissions: set the filename to create the experiment subimission

databases¶

--device-db

device database file (default: ‘“device_db.py”’)

Default: “device_db.py”

--dataset-db

dataset file (default: ‘“dataset_db.pyon”’)

Default: “dataset_db.pyon”

repository¶

-g, --git

use the Git repository backend

Default: False

-r, --repository

path to the repository (default: ‘“repository”’)

Default: “repository”

--experiment-subdir

path to the experiment folder from the repository root (default: ‘’)

Default: “”

logging¶

-v, --verbose

increase logging level of the master process

Default: 0

-q, --quiet

decrease logging level of the master process

Default: 0

--log-file

store logs in rotated files; set the base filename

Default: “”

--log-backup-count

number of old log files to keep, or 0 to keep all log files. ‘.<yyyy>-<mm>-<dd>’ is added to the base filename (default: 0)

Default: 0

artiq_client¶

ARTIQ CLI client

usage: artiq_client
       [-h]
       [-s SERVER]
       [--port PORT]
       [--version]
       {submit,delete,set-dataset,del-dataset,show,scan-devices,scan-repository,ls}
       ...

Positional Arguments¶

action: Possible choices: submit, delete, set-dataset, del-dataset, show, scan-devices, scan-repository, ls

Named Arguments¶

-s, --server

hostname or IP of the master to connect to

Default: “::1”

--port

TCP port to use to connect to the master

--version

print the ARTIQ version number

Sub-commands:¶

submit¶

submit an experiment

artiq_client submit
[-h]
[-p PIPELINE]
[-P PRIORITY]
[-t TIMED]
[-f]
[-R]
[-r REVISION]
[--content]
[-c CLASS_NAME]
FILE
[ARGUMENTS ...]

Positional Arguments¶

FILE: file containing the experiment to run
ARGUMENTS: run arguments

Named Arguments¶

-p, --pipeline

pipeline to run the experiment in (default: “main”)

Default: “main”

-P, --priority

priority (higher value means sooner scheduling, default: 0)

Default: 0

-t, --timed

set a due date for the experiment

-f, --flush

flush the pipeline before preparing the experiment

Default: False

-R, --repository

use the experiment repository

Default: False

-r, --revision

use a specific repository revision (defaults to head, ignored without -R)

--content

submit by content

Default: False

-c, --class-name

name of the class to run

delete¶

delete an experiment from the schedule

artiq_client delete
[-h]
[-g]
RID

Positional Arguments¶

RID: run identifier (RID)

Named Arguments¶

-g

request graceful termination

Default: False

set-dataset¶

add or modify a dataset

artiq_client set-dataset
[-h]
[-p | -n]
NAME
VALUE

Positional Arguments¶

NAME: name of the dataset
VALUE: value in PYON format

Named Arguments¶

-p, --persist

make the dataset persistent

Default: False

-n, --no-persist

make the dataset non-persistent

Default: False

del-dataset¶

delete a dataset

artiq_client del-dataset
[-h]
name

Positional Arguments¶

name: name of the dataset

show¶

show schedule, log, devices or datasets

artiq_client show
[-h]
WHAT

Positional Arguments¶

WHAT

Possible choices: schedule, log, ccb, devices, datasets

select object to show: [‘schedule’, ‘log’, ‘ccb’, ‘devices’, ‘datasets’]

scan-devices¶

trigger a device database (re)scan

artiq_client scan-devices
[-h]

scan-repository¶

trigger a repository (re)scan

artiq_client scan-repository
[-h]
[--async]
[REVISION]

Positional Arguments¶

REVISION: use a specific repository revision (defaults to head)

Named Arguments¶

--async

trigger scan and return immediately

Default: False

ls¶

list a directory on the master

artiq_client ls
[-h]
[directory]

Positional Arguments¶

directory: Default: “”

artiq_dashboard¶

ARTIQ Dashboard

usage: artiq_dashboard
       [-h]
       [--version]
       [-s SERVER]
       [--port-notify PORT_NOTIFY]
       [--port-control PORT_CONTROL]
       [--port-broadcast PORT_BROADCAST]
       [--db-file DB_FILE]
       [-p PLUGIN_MODULES]

Named Arguments¶

--version

print the ARTIQ version number

-s, --server

hostname or IP of the master to connect to

Default: “::1”

--port-notify

TCP port to connect to for notifications

Default: 3250

--port-control

TCP port to connect to for control

Default: 3251

--port-broadcast

TCP port to connect to for broadcasts

Default: 1067

--db-file

database file for local GUI settings

-p, --load-plugin

Python module to load on startup

artiq_session¶

ARTIQ session manager. Automatically runs the master, dashboard and local controller manager on the current machine. The latter requires the artiq-comtools package to be installed.

usage: artiq_session
       [-h]
       [--version]
       [-m M]
       [-d D]
       [-c C]

Named Arguments¶

--version

print the ARTIQ version number

-m

add argument to the master command line

Default: []

-d

add argument to the dashboard command line

Default: []

-c

add argument to the controller manager command line

Default: []