Provisioning & Configuration
Configuring on deployment
The transfer-agent supports resource provisioning via configuration file. This config can be passed by direct path, or by filename.
masv-agent -config=config.yml
If passed by filename, the agent will check the config directory:
/configfor the Docker image release~/.masvsrvfor the binary release
auth:
api_key: your-api-key
​settings:
upload_rate_limit:
enabled: true
rate_limit_bps: 10000000
schedule_enabled: true
schedule:
fri:
enabled: false
start: "09:00"
end: "17:00"
mon:
enabled: true
start: "09:00"
end: "17:00"
sat:
enabled: true
start: "09:00"
end: "17:00"
sun:
enabled: true
start: "09:00"
end: "17:00"
thu:
enabled: true
start: "09:00"
end: "17:00"
tue:
enabled: false
estart: "09:00"
end: "17:00"
wed:
enabled: false
start: "09:00"
end: "17:00"
download_rate_limit:
enabled: true
rate_limit_bps: 20000000
schedule_enabled: true
schedule:
mon:
enabled: true
start: "09:00"
end: "17:00"
fifo_mode:
enabled: true
active_download_count: 1
active_upload_count: 1
disk_write_mode:
calibration_mode: 0
manual_write_modes:
2049: 0
2050: 1
portal_download_automations:
- name: portal-download-auto
portal_subdomain: YourPortalSubdomain
destination_folder: /data/portal-downloads
create_package_folder: true
effective_time: 2021-07-30T00:00:00Z
enabled: true
priority: 0
all_portals_download_automations:
- name: all-portals-download-auto
team_id: YourTeamID
destination_folder: /data/portal-inbox
create_portal_subfolder: true
effective_time: 2023-09-25T00:00:00Z
enabled: true
priority: 3
portal_upload_automations:
- name: portal-upload-auto
portal_subdomain: YourPortalSubdomain
portal_password: PortalPassword
path: /data/watch1
sender_email: [email protected]
timeout: 5
delete_files_after_upload: false
send_loose_files: false
package_name_suffix: portal-suffix
message: portal upload msg
blacklist: ["*.png"]
enabled: true
priority: 2
team_upload_automations:
- name: team-upload-auto
team_id: TEAM_ID
path: /data/watch2
timeout: 5
recipient_emails: [[email protected]]
delete_files_after_upload: false
send_loose_files: false
package_name_suffix: PACKAGE_SUFFIX
message: team upload msg
download_limit: 7
delete_after: 3
download_password: DownloadPassword
blacklist: ["*.txt"]
enabled: false
priority: 1
unlimited_storage: false
tag:
id: tag_id
name: tag_name
portal_stream_automations:
- name: portal-stream-auto
portal_subdomain: subdomain
portal_password: password
path: /data/watch3
sender_email: [email protected]
timeout: 1
message: portal upload msg
whitelist: ["*.mxf"]
enabled: true
priority: 4
team_stream_automations:
- name: team-stream-auto
team_id: team_id
path: /data/watch4
timeout: 1
recipient_emails: [[email protected]]
message: team upload msg
download_limit: 7
delete_after: 2
download_password: asdfasdf
whitelist: ["*.mxf"]
enabled: false
priority: 3
Auth
The authentication method for the MASV API. Depending on the mechanism used, it can either have:
- An
api_keyproperty for API key based authentication - (Deprecated)
emailandpasswordproperties for user-based authentication
Settings
The following settings can be modified via the provisioning config file:
- Upload and download rate limit schedules
- FIFO mode settings
- Disk write mode (serial vs parallel)
Rate Limit Schedules
The rate limit schedules allow the app to automatically limit the transfer bandwidth during certain times and days of the week. The format for start and end is a HH:MM time stamp related to the local time. Days can be omitted, which will result in that day being disabled in the schedule.
FIFO Mode
FIFO mode settings, when enabled, determine the number of concurrent uploads/downloads that can be allocated bandwidth.
Disk Write Mode
Disk Write Mode settings determine whether the app will run in the standard parallel write mode, or serial write mode. In serial write mode only one write operation will be allowed at a time per disk.
If using the manual calibration mode (0), then the per disk write modes should be specified, keyed by the disk ID. For the per disk write modes, 0 indicates parallel, and 1 indicates serial.
The ID's for your disks can be found using the /api/v1/settings/disk_write_modes endpoint, and will be the keys in the response disk_write_modes objects.
{
"calibration_mode": 0,
"disk_write_modes": {
"2049": {
"disk_label": "/dev/sda1",
"requires_automatic_write_mode_calibration": true,
"write_mode_automatic": 0,
"write_mode_manual": 1
},
"2050": {
"disk_label": "/dev/sda2",
"requires_automatic_write_mode_calibration": true,
"write_mode_automatic": 0,
"write_mode_manual": 1
}
}
}
Portal Download Automations
Portal download automations will automatically poll the specified portal, and download all finalized packages that it finds created after the effective_time field.
All Portals Download Automations
All Portals download automations will automatically poll all portals associated with the specified team_id, downloading the finalized packages that it finds created after the effective_time field.
Upload Automations
Portal and Team upload automations both watch the specified parent folder for changes to the children. If no changes are detected in a child folder (or file, if send_loose_files=true) after timeout minutes, then the folder/file is uploaded to the portal/recipients.
Stream Automations
Portal and Team stream automations both watch the specified folder for new top level files. When a new file is detected, and optionally matches the whitelist, a new stream upload is started for that file. If timeout seconds pass without a change in file size, then the upload is finalized.