Provisioning & Configuration

Configuring on deployment

The transfer-agent supports resource provisioning via configuration file. This config can be passed by direct path, or by filename.

masv-agent -config=config.yml

If passed by filename, the agent will check the config directory:

• /config for the docker image release
• ~/.masvsrv for the binary release.

config.yaml

auth:
   api_key: your-api-key
   # email: test@test.com
   # password: topsecret

​settings:
  upload_rate_limit:
    enabled: true
    rate_limit_bps: 10000000
    schedule_enabled: true
    schedule:
      fri:
        enabled: false
          start: "09:00"
        end: "17:00"
      mon:
        enabled: true
        start: "09:00"
        end: "17:00"
      sat:
        enabled: true
        start: "09:00"
        end: "17:00"
      sun:
        enabled: true
        start: "09:00"
        end: "17:00"
      thu:
        enabled: true
        start: "09:00"
        end: "17:00"
      tue:
        enabled: false
        estart: "09:00"
        end: "17:00"
      wed:
        enabled: false
        start: "09:00"
        end: "17:00"

  download_rate_limit:
    enabled: true
    rate_limit_bps: 20000000
    schedule_enabled: true
    schedule:
      mon:
        enabled: true
        start: "09:00"
        end: "17:00"

  fifo_mode:
    enabled: true
    active_download_count: 1
    active_upload_count: 1

  disk_write_mode:
    calibration_mode: 0
    manual_write_modes:
      2049: 0
      2050: 1

portal_download_automations:
  - 1:
    name: portal-download-auto
    portal_subdomain: YourPortalSubdomain
    destination_folder: /data/portal-downloads
    create_package_folder: true
    effective_time: 2021-07-30T00:00:00Z
    enabled: true
    priority: 0

portal_upload_automations:
  - 1:
    name: portal-upload-auto
    portal_subdomain: YourPortalSubdomain
    portal_password: PortalPassword
    path: /data/watch
    sender_email: youremail@yourdomain.com
    timeout: 5
    delete_files_after_upload: false
    send_loose_files: false
    package_name_suffix: portal-suffix
    message: portal upload msg
    blacklist: ["*.png"]
    enabled: true
    priority: -1

team_upload_automations:
  - 1:
    name: team-upload-auto
    team_id: TEAM_ID
    path: /data/watch2
    timeout: 5
    recipient_emails: [recipient@example.com]
    delete_files_after_upload: false
    send_loose_files: false
    package_name_suffix: PACKAGE_SUFFIX
    message: team upload msg
    download_limit: 7
    delete_after: 3
    download_password: DownloadPassword
    blacklist: ["*.txt"]
    enabled: false
    priority: 1
    unlimited_storage: false
    tag:
      id:   tag_id
      name: tag_name

Auth

The authentication method for the MASV API. Depending on the mechanism used, it can either have:

• An api_key property for API key based authentication (recommended)
• email and password properties for user-based authentication (not recommended)

Settings

The following settings can be modified via the provisioning config file:

• upload and download rate limit schedules
• fifo mode settings
• disk write mode (serial vs parallel)

Rate Limit Schedules

The rate limit schedules allow the app to automatically limit the transfer bandwidth during certain times and days of the week. The format for start and end is a HH:MM time stamp related to the local time. Days can be omitted, which will result in that day being disabled in the schedule.

FIFO Mode

FIFO mode settings, when enabled, determine the number of concurrent uploads/downloads that can be allocated bandwidth.

Disk Write Mode

Disk Write Mode settings determine whether the app will run in the standard parallel write mode, or serial write mode. In serial write mode only one write operation will be allowed at a time per disk.

If using the manual calibration mode (0), then the per disk write modes should be specified, keyed by the disk ID. For the per disk write modes, 0 indicates parallel, and 1 indicates serial.

The ID's for your disks can be found using the /api/v1/settings/disk_write_modes endpoint, and will be the keys in the response disk_write_modes objects.

{
  "calibration_mode": 0,
  "disk_write_modes": {
    "2049": {
      "disk_label": "/dev/sda1",
      "requires_automatic_write_mode_calibration": true,
      "write_mode_automatic": 0,
      "write_mode_manual": 1
    },
    "2050": {
      "disk_label": "/dev/sda2",
      "requires_automatic_write_mode_calibration": true,
      "write_mode_automatic": 0,
      "write_mode_manual": 1
    }
  }
}

Portal Download Automations

Portal download automations will automatically poll the specified portal, and download all finalized packages that it finds that were created after the effective_time field.

Upload Automations

Portal and team upload automations both watch the specified parent folder for changes to the children. If no changes are detected in a child folder (or file, if send_loose_files=true) after timeout minutes, then the folder/file is uploaded to the portal/recipients.