Upload
MASV clients upload files directly to MASV's third-party storage service providers. MASV achieves this by providing pre-signed URL's for the chosen service while abstracting the interaction for each step of the file process.
Uploads examples are demonstrated using curl commands. If you have access to a terminal to run these commands, MASV recommends the use of our TransferAgent that will ease the process of integration.
Info
Once the upload package has been finalized, the transfer is locked and cannot be further modified.
Warning
Upload transfers created through the APIs expire (removed from the service and deleted) after the timeline set in your pricing subscription.
Packages and Authorization Mechanism
Files uploaded to MASV belong to a package object. This package can be thought of as a virtual directory inside of MASV. Each package contains a token that can be used to upload files to that specific virtual directory. A package can be created by interacting with a different API endpoint (for example, portals or teams).
All requests going to upload endpoints require a token passed as an HTTP header defined as X-Package-Token. This token allows the user to add/remove files from the package until the package is finalized.
Lifecycle Of An Upload
When uploading single or multiple files as part of a package, the following steps generalizes the upload interaction:
- For each file in the package.
- Add file to the package
API Action - Create the file in cloud storage
- Collect metadata from cloud storage service for the file
- Identify the number of file chunks to split the file
- Obtain authorized URL's to upload each chunk
- Upload all chunks to cloud storage
- Collect metadata from the cloud storage service for each uploaded chunk.
- Once all chunks are uploaded, send request to finalize file.
- Finalize package
Info
Once the upload package has been finalized, the transfer is locked and cannot be further modified.
Finalization of the package will then dispatch the file to the intended recipient immediately.
Warning
Please avoid uploading the following files/directories because they tend to change while upload is running which could cause the upload to fail:
desktop.ini.DS_Store.fcpcache
MASV Blueprints
MASV's API abstracts interactions with cloud storage services by creating a Blueprint object. This blueprint instructs the clients on which request they need to execute next during the file upload process.
Each blueprint has four properties:
| Name | Type | Required | Description |
|---|---|---|---|
url |
String | Yes | URL that the request should be forwarded towards. |
method |
String | Yes | The HTTP Method to use. Example: Get, POST, PUT, DELETE |
headers |
JSON | Yes | Key-value map of header names and their values. |
body |
String | Yes | HTTP Request body. |
Upload API Interaction
Each package belongs to either a team or portal on MASV. Depending on your use case you can create the initial package under either. Other API requests are portal/team agnostic.
1) Create a team/portal package
A. Team Package
| Method | Route |
|---|---|
POST |
/teams/{team_id}/packages |
HEADERS
| Name | Type | Required | Description |
|---|---|---|---|
X-User-Token |
String | Yes | User JSON Web Token |
Content-Type |
String | Yes | Must be application/json |
URL Parameters
| Name | Type | Required | Description |
|---|---|---|---|
team_id |
String | Yes | The team ID to bind the package to. |
BODY
| Name | Type | Required | Description |
|---|---|---|---|
access_limit |
Integer | No | Override default number of downloads for the package |
description |
String | Yes | Description of the package |
name |
String | Yes | Name of the package |
password |
String | No | Password to protect download access to the package |
recipients |
String[] | Yes | Email address of recipient(s) |
unlimited_storage |
Boolean | No | Enables and disables unlimited storage days for package |
B. Portal Package
| Method | Route |
|---|---|
POST |
/portals/{portal_id}/packages |
HEADERS
| Name | Type | Required | Description |
|---|---|---|---|
Content-Type |
String | Yes | Must be application/json |
X-Access-Code |
String | No | URI encoded upload password (if required by the Portal settings) |
URL Parameters
| Name | Type | Required | Description |
|---|---|---|---|
portal_id |
String | Yes | The portal id to bind the package to. (Retrieved in (Portals)) |
BODY
| Name | Type | Required | Description |
|---|---|---|---|
description |
String | Yes | Description of the package |
name |
String | Yes | Name of the package |
sender |
String | Yes | Email address of the portal package sender |
Warning
If you set unlimited_storage to true, Unlimited Storage will be activated and Unlimited Storage charges will be incurred daily beyond the free period included with the team subscription for users who have activated their account. Trial accounts cannot extend the expiry beyond the original date given on creation - it can only be set earlier. How does Extended Storage Work?
REQUEST
curl -d "{\"access_limit\":$ACCESS_LIMIT, \"description\":\"$DESCRIPTION\", \"name\":\"$NAME\", \"password\": \"$PASSWORD\", \"recipients\":[\"$RECIPIENT_EMAIL\"]}" \
-H "X-User-Token: $USER_TOKEN" \
-H "Content-Type: application/json" \
-s -X POST https://api.massive.app/v1/teams/$TEAM_ID/packages
After a successful request where a package has been created, this endpoint will return an HTTP response with a status code of 201 Created and a body similar to the one below.
{
"access_token": "xxxxxxxxxxxxxxxxx",
"created_at": "2021-07-26T18:41:24.508Z",
"custom_expiry": true,
"unlimited_storage": false,
"description": "Testing 123",
"expiry": "2021-08-05T18:41:24.508Z",
"extra_storage_updated_at": "2021-07-26T18:41:24.508Z",
"extras": {
"storage_id": "aws-wdc"
},
"id": "01FBJ1Y9GW3DQ1VHBAJZ7J49FQ",
"name": "File",
"progress_channels": [
{
"auth": {
"auth_key": "xxxxxxxxxxxxxxxxx",
"channel_name": "proc.upload.01D72PEE5AH74NR9EG4CZS29JZ",
"publish_key": "pub-c-xxxxx-xxxxxxxxxxxxx",
"subscribe_key": "sub-c-xxxxx-xxxxxxxxxxxxx"
},
"topic":"upload",
"provider": "pubnub"
}
],
"recipients": [
"[email protected]"
],
"state": "new",
"updated_at": "2021-07-26T18:41:24.508Z",
"usage_updated_at": "2021-07-26T18:41:24.508Z"
}
Response Properties:
| Property | Description |
|---|---|
access_token |
The auth token returned during auth request. |
created_at |
Time that the object was created at |
custom_expiry |
Was a custom expiration set |
description |
The description for the package. |
expiry |
The expiry for the package |
extra_storage_updated_at |
The time the extra storage was updated at. |
extras |
Various extra fields that can be populated by the user |
extras.storage_id |
The storage id for extras. |
id |
The unique package identifier. |
name |
The name of the package |
progress_channels |
PubNub progress events |
recipients |
Array of emails to send the package to. |
state |
Current state of the package |
updated_at |
Last time the package was updated. |
usage_updated_at |
When usage was last calculated. |
Note
Take note of the id and the access_token as they are required to interact with the upload API.
2) Add a file to the package
| Method | Route |
|---|---|
POST |
/packages/{package_id}/files |
HEADERS
| Name | Type | Required | Description |
|---|---|---|---|
X-Package-Token |
String | Yes | Package JSON Web Token |
Content-Type |
String | Yes | Must be application/json |
BODY
| Name | Type | Required | Description |
|---|---|---|---|
kind |
String | Yes | Type of file Accepted Values: file/directory |
name |
String | Yes | Filename (in the file system) |
path |
String | Yes | File's relative path (in the file system) |
last_modified |
String | Yes | File's last modified date in the file system (in UTC format) |
REQUEST PARAMETERS
| Name | Type | Required | Description |
|---|---|---|---|
package_id |
String | Yes | Package ID obtained in step 1 |
Info
For the following request we will use a specific file example:
- Kind: file
- Name: my_video.mpeg
- Path: Same as where the curl command is being executed
- Size: 210 MiB in size
REQUEST
curl -d '{"kind":"file", "name":"my_video.mpeg", "path": "", "last_modified":"2018-12-17T16:14:34.450Z"}' \
-H "X-Package-Token: $PACKAGE_TOKEN" \
-H "Content-Type: application/json" \
-s -X POST https://api.massive.app/v1/packages/$PACKAGE_ID/files
After a successful request where a file has been added to the package, this endpoint will return an HTTP response with a status code of 201 Created and a body similar to the one below.
{
"storage_type": "s3",
"max_chunk_size": 5368709120,
"max_chunks_count": 10000,
"create_blueprint": {
"headers": {
"Content-Disposition": "attachment; filename=\"my_video.mpeg\"; filename*=UTF-8my_video.mpeg"
},
"method": "POST",
"url": "https://masv3-storage-prod-wdc.s3-accelerate.amazonaws.com/01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJT7DTCC6DRVEACYQ%2F20190214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190214T002448Z&X-Amz-Expires=172799&X-Amz-SignedHeaders=host&uploads=&X-Amz-Signature=4bc45729a1e768a04b655f4a5972b548db1f8b359be16e719434bc5cc7ffd019"
},
"file": {
"id": "01D3MP8G3RBA7A1P0K98VEV2V3",
"last_modified": "2018-12-17T16:14:34.450Z",
"name": "my_video.mpeg"
}
}
Response Properties:
| Name | Type | Required | Description |
|---|---|---|---|
storage_type |
String | Yes | Type of cloud storage service the file should be uploaded to. Required value: s3 |
max_chunk_size |
String | Yes | Maximum allowed chunk size by the cloud service. |
max_chunks_count |
String | yes | Maximum number of chunks allowed for the file. |
create_blueprint |
Blueprint{} | yes | The request blueprint for creating the file. |
file |
file{} | yes | File metadata, take note of property: file.id as it is needed later. |
3) Create the file in MASV's cloud storage
From Step 2, use the create_blueprint and construct a corresponding HTTP request based on the supplied information. For example, using the create_blueprint above, one can construct a curl request as follows:
curl -H "BLUE_PRINT_HDR" \
-s -X BLUE_PRINT_METHOD 'BLUE_PRINT_URL'
Where:
HEADERS
| Name | Type | Required | Description |
|---|---|---|---|
BLUE_PRINT_HDR |
String | Yes | property: headers from CreateBlueprint step. |
REQUEST PARAMETERS
| Name | Type | Required | Description |
|---|---|---|---|
BLUE_PRINT_METHOD |
String | Yes | property: method from CreateBlueprint step. |
BLUE_PRINT_URL |
String | Yes | property: url from CreateBlueprint step. |
The cloud storage service (in this case Amazon AWS S3) response will be similar to the following:
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Bucket>masv3-storage-prod-wdc</Bucket>
<Key>01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3</Key>
<UploadId>Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI</UploadId>
</InitiateMultipartUploadResult>
Note
Data format in the response will be XML.
Note
Ensure to store property: uploadId from Step 3 as it is metadata required for interacting with the upload API methods related to this file.
4) Obtain upload URLs
This endpoint is used to get pre-signed upload URLs for each of a file's parts. These upload URLs are essentially limited access to storage.
Note
Upload URLs are valid for a limited amount of time and must be re-requested if they expire.
The client can request upload URLs in bulk which will save a lot of back-and-forth communication with the MASV API.
POST /packages/{package_id}/files/{file_id}
HEADERS
| Name | Type | Required | Description |
|---|---|---|---|
X-Package-Token |
String | Yes | Package JSON Web Token |
Content-Type |
String | Yes | Must be application/json |
BODY
| Name | Type | Required | Description |
|---|---|---|---|
upload_id |
String | Yes | ID for the initiated multi-part upload |
PARAMETERS
| Name | Type | Required | Description |
|---|---|---|---|
package_id |
String | Yes | Package ID obtained in step 1 |
file_id |
string | yes | File ID obtained from step 1. |
start |
Integer | Yes | Starting index of the chunk. (zero-indexed) |
count |
Integer | Yes | Number of chunk requests to generate |
REQUEST
curl -d "{\"upload_id\":\"$UPLOAD_ID\"}" \
-H "X-Package-Token: $PACKAGE_TOKEN" \
-H "Content-Type: application/json" \
-s -X POST https://api.massive.app/v1/packages/$PACKAGE_ID/files/$FILE_ID?start=$START&count=$COUNT
Our example requires 3 blueprints (for the 3 chunks it's trying to upload START=0 and COUNT=3).
The start and count parameters provide control over how many bulk chunk requests are obtained. This is useful for times when total file size upfront is unavailable (ie: data streams.)
The server will return an array of blueprints equal to parameter: count. The blueprints are in ascending order based on the chunk index (starting at 0.)
To continue with our example of uploading my_video.mpeg. We would receive the following output:
[
{
"method": "PUT",
"url": "https://masv3-storage-prod-wdc.s3-accelerate.amazonaws.com/01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJT7DTCC6DRVEACYQ%2F20190214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190214T011216Z&X-Amz-Expires=86399&X-Amz-SignedHeaders=host&partNumber=1&uploadId=Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI&X-Amz-Signature=a1ac9186728c3b45d21afc48a70d1535f7597b4f885b0c6ce8cafbbdcde72c72"
},
{
"method": "PUT",
"url": "https://masv3-storage-prod-wdc.s3-accelerate.amazonaws.com/01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJT7DTCC6DRVEACYQ%2F20190214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190214T011216Z&X-Amz-Expires=86399&X-Amz-SignedHeaders=host&partNumber=2&uploadId=Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI&X-Amz-Signature=fba71b0a4eaf033ac47c12aaaf310cba5f78280b75e44cb2176d9f3b0e64b794"
},
{
"method": "PUT",
"url": "https://masv3-storage-prod-wdc.s3-accelerate.amazonaws.com/01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJT7DTCC6DRVEACYQ%2F20190214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190214T011216Z&X-Amz-Expires=86399&X-Amz-SignedHeaders=host&partNumber=3&uploadId=Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI&X-Amz-Signature=0e6c68340af207d43efe23812e831ccc93a0f83a2bbf9ca0181051b9798cd6d8"
}
]
5) Upload the file chunks
We now begin with the actual upload of your file. Using the blueprint received you can upload each chunk as binary data in the request body.
Continuing with our my_video.mpeg upload. We want to upload chunks of 100mb, dd reads in chunk data as bytes. Using a simple converter we can convert 100mb to 104857600 bytes. We input this into our curl command to tell it how many bytes need to be read.
Info
Note: The <() is a bit of bash magic to read a byte range of a file without having to physically chunk it
curl -i -s -X BLUE_PRINT_METHOD "BLUE_PRINT_URL" –-data-binary @FILE_CHUNK,
curl -i -s -X PUT BLUEPRINT_URL \
--data-binary @<(dd if=my_video.mpeg skip=0 count=104857600 iflag=skip_bytes,count_bytes)
Where:
| Name | Type | Required | Description |
|---|---|---|---|
BLUE_PRINT_METHOD |
String | Yes | parameter: method obtained in step 5 |
BLUE_PRINT_URL |
String | Yes | parameter: url obtained in step 5 |
FILE_CHUNK |
String | yes | The file chunk to be uploaded |
We would receive the following response:
Content-Length: 0
Connection: keep-alive
x-amz-id-2: NgcsGsD8A5YSO+uNfp5plgERV1Y1uPuAAz2mHSQB++1g/MdNBYh6kXrK7zkVHzOxhzSmv4jAu1w=
x-amz-request-id: 65CAB8D270C820CB
Date: Thu, 14 Feb 2019 01:47:39 GMT
ETag: "7be1b3cc95a04a6dd07d157ad6fae64a"
Server: AmazonS3
X-Cache: Miss from cloudfront
Via: 1.1 56f9e0effaf2e1930aee6248ba70abb7.cloudfront.net (CloudFront)
X-Amz-Cf-Id: giSjIOdkcJ4qbIAcPBq--uItpHaRXb-8ER4YUjvQcx-yKTUOjNRQ4w==
Note
For each uploaded chunk, the following metadata must be collected:
partNumberwhich is part of the chunk upload request URL parametersETagwhich is returned as a response header by the storage service
6) Publish upload progress
Note
This feature is only available to customers with this feature enabled. To request access to upload progress API and channels for your team, please contact team@masv.io.
MASV integrates with PubNub to indicate overall file upload progress on each package. These progress updates are posted to the progress channel returned in the Create Package response.
Using the credentials returned for the upload topic in the progress_channels property of the response, messages should be publishing following this format:
{
"completed": 123544,
"total": 1073741824,
"speed_bps": 123,
"eta": 301.054
}
Properties:
| Name | Type | Required | Description |
|---|---|---|---|
completed |
String | Yes | Current number of bytes successfully uploaded. |
total |
String | Yes | Total size of all files in package that will be uploaded format: bytes |
speed_bps |
String | Yes | Rolling average of bits per second |
eta |
String | Yes | Estimated time of arrival based on speed_bps in seconds. |
Note
For more information on integrating PubNub into your client/service refer to their extensive developer documentation..
7) Finalize the file
Once all chunks have been uploaded, the client must inform the API that the file is successfully uploaded to the cloud storage service. This call informs the MASV backend that all uploading for the file is completed and must supply all collected metadata needed to seal the file in storage.
| Method | Route |
|---|---|
POST |
/packages/{package_id}/files/{file_id}/finalize |
HEADERS
| Name | Type | Required | Description |
|---|---|---|---|
X-Package-Token |
String | Yes | Package JSON Web Token |
Content-Type |
String | Yes | Must be application/json |
BODY
| Name | Type | Required | Description |
|---|---|---|---|
chunk_extras |
Array Of Objects | Yes | Information about all the chunks that make up the file |
chunk_extras[$index].partNumber |
String | Yes | Chunk part number |
chunk_extras[$index].etag |
String | Yes | Chunk part hash |
chunk_extras[$index].file_extras |
Array of Objects | Yes | Chunk part extras |
file_extras.uploadId |
String | Yes | The ID of the upload provided in from step 3 |
size |
Integer | Yes | The file size |
chunk_size |
Integer | * | The chunk size used when uploading the file. There is a performance impact if this field is not provided. Providing this field ensures the generation of a manifest file used to verify file integrity. In custom upload implementations, this field should always be provided. |
REQUEST
curl -d '{"chunk_extras":[{"partNumber": CHUNK_PART_NUMBER, "etag": CHUNK_ETAG}], "file_extras":{"upload_id": UPLOAD_ID}, "size": FILE_SIZE, "chunk_size": CHUNK_SIZE}' \
-H "X-Package-Token: PACKAGE_TOKEN" \
-H "Content-Type: application/json" \
-s -X POST https://api.massive.app/v1/packages/PACKAGE_ID/files/FILE_ID/finalize
For our my_video.mpeg example, our JSON body would be similar to below:
{
"size": 147778580,
"file_extras": {
"upload_id": "Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI"
},
"chunk_extras": [
{
"partNumber": "1",
"etag": "\"7be1b3cc95a04a6dd07d157ad6fae64a\""
},
{
"partNumber": "2",
"etag": "\"ca8e85fc588f90d07449d0303e9cf329\""
},
{
"partNumber": "3",
"etag": "\"a6a6aa6836a7a70c28a06dec02dbf199\""
}
]
}
On success, the server will respond with an empty body and a 204 No Content status code.
Note
The client has to repeat steps 2-7 for each additional file in the package.
Attention
If the file size submitted in this request does not match the actual size of the object in storage the service will respond with a 409 Conflict status code.
8) Finalize the package
Once all files have been uploaded and finalized, the client must indicate that the package has been completed and ready to be dispatched to the intended recipient.
Attention
Make sure all uplaoded files have been completed and finalized before finalizing the package. Any in-flight, non-finalized files will be dropped when finalize package is called. If no files were uploaded/finalized, this API call will fail.
This call informs that MASV backend that package is ready to be dispatched to the intended recipient. The package will be dispatched to the intended recipient immediately upon the finalize call.
POST /packages/{package_id}/finalize
Note
Once a package is finalized, its expiry and link(s) expiry time are extended by the amount of time the upload took.
HEADERS
| Name | Type | Required | Description |
|---|---|---|---|
X-Package-Token |
String | Yes | Package JSON Web Token |
Content-Type |
String | Yes | Must be application/json |
URL Parameters
| Name | Type | Required | Description |
|---|---|---|---|
package_id |
String | yes | Package ID obtained in step 1 |
REQUEST
curl -H "X-Package-Token: $PACKAGE_TOKEN" \
-H "Content-Type: application/json" \
-s -X POST https://api.massive.app/v1/packages/PACKAGE_ID/finalize
On success, the server will respond with an empty body and a 204 No Content status code.
Creating Additional Links
MASV's API allows you to create additional direct-download links or send a link to a specific email recipient after the upload has been finalized. See the Links page for more details and examples.