Skip to content

Upload

MASV clients upload files directly to MASV's third-party storage service provider (such as Microsoft Azure Blob Storage or Amazon AWS S3). The MASV API supplies authorized links to clients in order to carry out the uploads to these storage services. This is achieved by providing pre-signed URL's for each specific service and abstracting the interaction needed for each step of the file upload process.

Upload transfers created through the APIs expire after 10 days. A transfer request consists of the endpoint itself, the headers, and the body, which can contain a message and must contain a list of file objects. Clients MUST supply the API with file metadata (name, modified date, path and size) while uploading the file contents to cloud storage.

Once the upload package has been finalized, the transfer is locked and cannot be further modified.

Packages and Authorization Mechanism

Each file uploaded to MASV cloud storage belongs to a package. A package can be created by interacting with a different API endpoint (for example, portals or teams). Once a package is created, the client obtains a package token that can be used to upload files for that specific package.

The package token must be passed with any requests going to the upload API endpoints by means of an HTTP header called X-Package-Token. This token gives the client full ownership over the package, so it can add/remove files from the package or remove the package entirely - until the package is finalized.

Lifecycle Of An Upload

When uploading single or multiple files as part of a package, the following steps generalizes the upload interaction:

  • Create package
  • For each file in the package
    • Add file to the package obtain storage request blueprint to create file in MASV's third party storage service
    • Create the file in cloud storage
    • Collect metadata returned by the cloud storage service for the created file
    • Identify the number of chunks the file will be split up into
    • Obtain authorized URL's to upload each chunk
    • Upload all chunks to cloud storage
    • Collect metadata returned by the cloud storage service for each uploaded chunk
    • Once all chunks are uploaded, send a request to API to finalize the file
  • Finalize package

Note

Once a file is finalized, no more chunks can be added to it and it cannot be modified.

Note

Once a package is finalized, no more files can be added or deleted from the package. The package will be dispatched to the intended recipient immediately upon the finalize call.

MASV Blueprints

MASV's API abstracts the interaction with cloud storage services by instructing the clients on which request they need to execute next during the file upload process. This abstraction is defined by a createBlueprint data structure, which holds all the information required to represent an HTTP request. Each blueprint has four properties (some optional):

  • url which is the URL to which the request should be forwarded. This includes the scheme, host, path and parameters
  • method which identifies the HTTP method type like GET, POST, PUT or DELETE
  • headers which is a key-value map of header names and their corresponding values that need to accompany the request
  • body is the request's body

Upload API Interaction

1) Create a team package

POST /teams/{team_id}/packages

HEADERS
Name Type Required Description
X-User-Token String Yes User JSON Web Token
Content-Type String Yes Must be application/json
BODY
Name Type Required Description
access_limit Integer No Override default number of downloads for the package
description String Yes Description of the package
name String Yes Name of the package
password String No Password to protect download access to the package
recipients Array Of Strings Yes Email address of recipient(s)
REQUEST
curl -H 'X-User-Token: $USER_TOKEN ' -H 'Content-Type: application/json' -s -X POST https://api.massive.app/v1/teams/$TEAM_ID/packages  -d '{"access_limit":"$ACCESS_LIMIT", "description":"$DESCRIPTION" "name":"$NAME", "password": "$PASSWORD", "recipients":["$R1_EMAIL","$R2_EMAIL"]}'

Where:

  • $USER_TOKEN is the auth token returned during auth request (refer to Authorization: Login section of this document)
  • $TEAM_ID is the team ID returned during auth request (refer to Authorization: Login section of this document)

After a successful request where a package has been created, this endpoint will return an HTTP response with a status code of 201 Created and a body similar to the one below.

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ8.eyJleHAiOjE1NTQzOTk4NzAsImltcCI6ZmFsc2UsImx2bCI6InUiLCJzdWIiOiIwMUQ3MlBFRTVBSDc0TlI5RUc0Q1pTMjlKVyIsInR5cCI6InBhY2thZ2UiLCJ1aWQiOiIwMUQzOThLV1NYV1M4WUJaS1hDVzU2QkY0SCJ9.ko0MKECmwu12LIQgyPOCAGx1WtaBPyhwcBdF41DdhvE",
  "completed": false,
  "created_at": "2019-03-28T17:44:30.122Z",
  "description": "Hello world",
  "expiry": "2019-04-07T17:44:30.122Z",
  "extras": {
    "storage_id": "aws-wdc"
  },
  "id": "01D72PEE5AH74NR9EG4CZS29JZ",
  "name": "Test package",
  "recipients": [
    "[email protected]",
    "[email protected] "
  ],
  "updated_at": "2019-03-28T17:44:30.122Z",
  "usage_updated_at": "2019-03-28T17:44:30.122Z"
}

Note

Take note of the id and the access_token as they are required to interact with the upload API.

2) Add a file to the package

POST /packages/{package_id}/files

HEADERS
Name Type Required Description
X-Package-Token String Yes Package JSON Web Token
Content-Type String Yes Must be application/json
BODY
Name Type Required Description
kind String No Type of file: file or directory
name String Yes Filename (in the file system)
path String Yes File's relative path (in the file system)
last_modified String Yes File's last modified date in the file system (in UTC format)

Info

For the following request we will use a specific file example:

  • Kind: file
  • Name: my_video.mpeg
  • Path: Same as where the curl command is being executed
  • Size: 210 MiB in size
REQUEST
curl -H 'X-Package-Token: $PACKAGE_TOKEN ' -H 'Content-Type: application/json' -s -X POST https://api.massive.app/v1/packages/$PACKAGE_ID/files  -d '{"kind":"file", "name":"my_video.mpeg", "path": "", "last_modified":"2018-12-17T16:14:34.450Z"}'

Where:

  • $PACKAGE_TOKEN is the access token obtained from Step 1
  • $PACKAGE_ID is the id obtained from Step 1

After a successful request where a file has been added to the package, this endpoint will return an HTTP response with a status code of 201 Created and a body similar to the one below.

{
  "storageType": "s3",
  "maxChunkSize": 5368709120,
  "maxChunksCount": 10000,
  "createBlueprint": {
    "headers": {
      "Content-Disposition": "attachment; filename=\"my_video.mpeg\"; filename*=UTF-8my_video.mpeg"
    },
    "method": "POST",
    "url": "https://masv3-storage-prod-wdc.s3-accelerate.amazonaws.com/01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJT7DTCC6DRVEACYQ%2F20190214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190214T002448Z&X-Amz-Expires=172799&X-Amz-SignedHeaders=host&uploads=&X-Amz-Signature=4bc45729a1e768a04b655f4a5972b548db1f8b359be16e719434bc5cc7ffd019"
  },
  "file": {
    "id": "01D3MP8G3RBA7A1P0K98VEV2V3",
    "last_modified": "2018-12-17T16:14:34.450Z",
    "name": "my_video.mpeg"
  }
}

Where:

  • storageType is the type of cloud storage service the file should be uploaded to. Currently this can be s3 or azure. The value of this property dictates the behaviour of metadata collection after each direct interaction with the storage.
  • maxChunkSize is the maximum allowed chunk size by the cloud storage service (in bytes)
  • maxChunkCount is the maximum number of chunks allowed for the file
  • createBlueprint is the request blueprint for creating the file in storage
  • file is the file metadata. Take note of the file.id value as it will be needed to interact later with the upload API.

3) Create the file in MASV's cloud storage

From Step 2, use the createBlueprint and construct a corresponding HTTP request based on the supplied information. For example, using the createBlueprint above, one can construct a curl request as follows:

curl -H "$BLUE_PRINT_HDR" -s -X $BLUE_PRINT_METHOD '$BLUE_PRINT_URL'

Where:

  • $BLUE_PRINT_HDR is the headers value returned in create createBlueprint obtained in Step 2
  • $BLUE_PRINT_METHOD is the method value returned in create createBlueprint obtained in Step 2
  • $BLUE_PRINT_URL is the url value returned in create createBlueprint obtained in Step 2

The cloud storage service (in this case Amazon AWS S3) response will be similar to the following:

<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Bucket>masv3-storage-prod-wdc</Bucket>
    <Key>01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3</Key>
    <UploadId>Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI</UploadId>
</InitiateMultipartUploadResult>

Note

Data format in the response will be XML.

Note

Ensure to store uploadId from Step 3 as it is metadata required for interacting with the upload API methods related to this file.

4) Split up the file for upload

To be able to upload a file, it must be split into parts and then each part will be uploaded to pre-signed URLs. These upload URLs are essentially limited access to a storage. At this point, the client should decide on a chunking strategy for the file. The chunk size should not exceed the maxChunkSize from Step 2, and the total number of chunks per file should not exceed the maxChunkCount. MASV's web and desktop client use a minimum chunk size of 100 MiB, which is adapted for larger files so the chunk count never exceeds maxChunkCount.

For our example file (which is 210 MiB in size), we'll use a 100 MiB chunk size so it will be split up into three chunks (2x100 MiB chunks and 1x10 MiB chunk).

Chunking in and of itself does not have to be physical, but rather logical. The client just needs to decide on which byte range of the file belongs to each chunk. Depending on the environment, this might not be strictly possible, but most modern filesystems allow random read access to any file. Please note that if the file is too small, then it will be uploaded as a single chunk file.  

5) Obtain upload URLs

This endpoint is used to get pre-signed upload URLs for each of a file's parts. These upload URLs are essentially limited access to storage.

Note

Upload URLs are valid for a limited amount of time and must be re-requested if they expire.

The client can request upload URLs in bulk which will save a lot of back-and-forth communication with the MASV API.

POST /packages/{package_id}/files/{file_id}

HEADERS
Name Type Required Description
X-Package-Token String Yes Package JSON Web Token
Content-Type String Yes Must be application/json
BODY
Name Type Required Description
uploadId String Yes ID for the initiated multi-part upload
QUERY
Name Type Required Description
start Integer Yes Starting index of the chunk
count Integer Yes Number of chunk requests to generate
REQUEST
curl -H 'X-Package-Token: $PACKAGE_TOKEN ' -H 'Content-Type: application/json' -s -X POST https://api.massive.app/v1/packages/$PACKAGE_ID/files/$FILE_ID?start=$START&count=$COUNT  -d '{"uploadId":"$UPLOAD_ID"}'

Where:

  • $PACKAGE_ID is the id obtained from Step 1
  • $PACKAGE_TOKEN is the access token obtained from Step 1
  • $FILE_ID is the id obtained from Step 3
  • $START is the start index (zero-indexed) of the chunk
  • $COUNT is the number of chunk requests to generate
  • $UPLOAD_ID is the upload ID (uploadId) obtained from Step 4

Using our example, the client requires 3 blueprints (for the 3 chunks it's trying to upload $START=0 and $COUNT=3).

The start and count parameters give clients control over how many bulk chunk requests they want to obtain. This is particularly useful for clients that do not know the total file size upfront (such as data streams for example).

The server will respond with an array of blueprints. The array size will be equal to the count parameter. The array elements are ordered so that the first blueprint should be used for chunk at index start, which is 0 in this case.

Continuing with our my_video.mpeg example, the output should be similar to the following:

[
  {
    "method": "PUT",
    "url": "https://masv3-storage-prod-wdc.s3-accelerate.amazonaws.com/01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJT7DTCC6DRVEACYQ%2F20190214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190214T011216Z&X-Amz-Expires=86399&X-Amz-SignedHeaders=host&partNumber=1&uploadId=Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI&X-Amz-Signature=a1ac9186728c3b45d21afc48a70d1535f7597b4f885b0c6ce8cafbbdcde72c72"
  },
  {
    "method": "PUT",
    "url": "https://masv3-storage-prod-wdc.s3-accelerate.amazonaws.com/01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJT7DTCC6DRVEACYQ%2F20190214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190214T011216Z&X-Amz-Expires=86399&X-Amz-SignedHeaders=host&partNumber=2&uploadId=Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI&X-Amz-Signature=fba71b0a4eaf033ac47c12aaaf310cba5f78280b75e44cb2176d9f3b0e64b794"
  },
  {
    "method": "PUT",
    "url": "https://masv3-storage-prod-wdc.s3-accelerate.amazonaws.com/01D3MNHA6P4XFXDMHJDTXT61CB/01D3MP8G3RBA7A1P0K98VEV2V3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJT7DTCC6DRVEACYQ%2F20190214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190214T011216Z&X-Amz-Expires=86399&X-Amz-SignedHeaders=host&partNumber=3&uploadId=Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI&X-Amz-Signature=0e6c68340af207d43efe23812e831ccc93a0f83a2bbf9ca0181051b9798cd6d8"
  }
]

6) Upload the file chunks

Time to actually upload (chunks of) your file. With the chunk upload blueprints at hand, the client is now responsible for uploading each chunk by executing the corresponding request blueprint, but this time using the chunk binary data as request body.

curl -i -s -X $BLUE_PRINT_METHOD '$BLUE_PRINT_URL' –-data-binary @$FILE_CHUNK",

Where:

  • $BLUE_PRINT_METHOD is the method value returned in createBlueprint obtained in Step 5
  • $BLUE_PRINT_URL is the url value returned in createBlueprint obtained in Step 5
  • $FILE_CHUNK is the file chunk to be uploaded

For example, to upload the chunks of our example file, we'll have to execute three requests (one for each chunk). Note that the <() syntax is bash magic to read a byte range of a file without having to physically chunk it. Also, notice the -i flag for curl which prints out he response headers:

curl -i -s -X PUT $BLUEPRINT_URL \
  --data-binary @<(dd if=my_video.mpeg skip=0 count=104857600 iflag=skip_bytes,count_bytes)

The output would look similar to the following:

Content-Length: 0
Connection: keep-alive
x-amz-id-2: NgcsGsD8A5YSO+uNfp5plgERV1Y1uPuAAz2mHSQB++1g/MdNBYh6kXrK7zkVHzOxhzSmv4jAu1w=
x-amz-request-id: 65CAB8D270C820CB
Date: Thu, 14 Feb 2019 01:47:39 GMT
ETag: "7be1b3cc95a04a6dd07d157ad6fae64a"
Server: AmazonS3
X-Cache: Miss from cloudfront
Via: 1.1 56f9e0effaf2e1930aee6248ba70abb7.cloudfront.net (CloudFront)
X-Amz-Cf-Id: giSjIOdkcJ4qbIAcPBq--uItpHaRXb-8ER4YUjvQcx-yKTUOjNRQ4w==

Note

For each uploaded chunk, the following metadata must be collected:

  • partNumber which is part of the chunk upload request URL parameters
  • ETag which is returned as a response header by the storage service

7) Finalize the file

Once all chunks have been uploaded, the client must tell the API that the file has been successfully uploaded to the cloud storage service and supply all collected metadata needed by the API to seal the file in storage.

This call informs your the MASV backend that all the uploading for your file is done.

POST /packages/{package_id}/files/{file_id}/finalize

HEADERS
Name Type Required Description
X-Package-Token String Yes Package JSON Web Token
Content-Type String Yes Must be application/json
BODY
Name Type Required Description
chunkExtras Array Of Objects Yes Information about all the chunks that make up the file
REQUEST
curl -H 'X-Package-Token: $PACKAGE_TOKEN ' -H 'Content-Type: application/json' -s -X POST https://api.massive.app/v1/packages/$PACKAGE_ID/files/$FILE_ID/finalize -d {"chunkExtras":[{"partNumber": $CHUNK_PART_NUMBER, "etag": $CHUNK_ETAG}], "fileExtras":[{"uploadId": $UPLOAD_ID}], "size": $FILE_SIZE}'

Where:

  • $PACKAGE_ID is the id obtained from Step 1
  • $PACKAGE_TOKEN is the access token obtained from Step 1
  • $FILE_ID is the id obtained from Step 3
  • $UPLOAD_ID is the upload ID (uploadId) obtained from Step 4
  • chunkExtras is an array of metadata collected in Step 6, which corresponds to each chunk upload's partNumber from request URL parameter and ETag header value from

Continuing with our specific, the request body would be:

{
  "size": 147778580,
  "fileExtras": {
    "uploadId": "Gng8Oh.yKzM41CmGxMRzHkO20GbP2RarZfTJVHHyPGqTVwrHHRgIZKqxMykJBjzDLZ9o3.pyYPwbVbtSI2LDOEa96Mllx6roowA47KHu2GkXcH6PC7qJTFDtYS38SwqI"
  },
  "chunkExtras": [
    {
      "partNumber": "1",
      "etag": "\"7be1b3cc95a04a6dd07d157ad6fae64a\""
    },
    {
      "partNumber": "2",
      "etag": "\"ca8e85fc588f90d07449d0303e9cf329\""
    },
    {
      "partNumber": "3",
      "etag": "\"a6a6aa6836a7a70c28a06dec02dbf199\""
    }
  ]
}

On success, the server will respond with an empty body and a 204 No Content status code.

Note

The client has to repeat steps 2-7 for each additional file in the package.

8) Finalize the package

Once all files have been uploaded, the client must indicate that the package has been completed and ready to be dispatched to the intended recipient.

This call informs that MASV backend that package is ready to be dispatched to the intended recipient. The package will be dispatched to the intended recipient immediately upon the finalize call.

POST /packages/{package_id}/finalize

HEADERS
Name Type Required Description
X-Package-Token String Yes Package JSON Web Token
Content-Type String Yes Must be application/json
REQUEST
curl -H 'X-Package-Token: $PACKAGE_TOKEN ' -H 'Content-Type: application/json' -s -X POST https://api.massive.app/v1/packages/$PACKAGE_ID/finalize

Where:

  • $PACKAGE_ID is the id obtained from Step 1
  • $PACKAGE_TOKEN is the access token obtained from Step 1

On success, the server will respond with an empty body and a 204 No Content status code.

MASV's API allows you to create additional direct-download links or send a link to a specific email recipient after the upload has been finalized. To create a link the package must be in the "finalized" state; links cannot be created while the upload is in progress.

POST /packages/{package_id}/links

HEADERS
Name Type Required Description
X-Package-Token String Yes Package JSON Web Token
Content-Type String Yes Must be application/json
BODY
Name Type Required Description
email String No Recipient email address or empty for direct-download link
password String No Password that user must enter before being able to view and download the package files
REQUEST
curl -H 'X-Package-Token: $PACKAGE_TOKEN ' -H 'Content-Type: application/json' -s -X POST https://api.massive.app/v1/packages/$PACKAGE_ID/links  -d '{"email":"[email protected]", "password":"somepassword"}'

Where:

  • $PACKAGE_TOKEN is the access token obtained from Step 1 of uploading the package
  • $PACKAGE_ID is the id obtained from Step 1 of uploading the package

After a successful request where a link has been added to the package, this endpoint will return an HTTP response with a status code of 201 Created and a body similar to the one below.

{
    "access_limit": 5,
    "access_limit_enabled": true,
    "active": true,
    "download_secret": "bqkBtnPwvmkgJpHN",
    "email": "[email protected]",
    "expiry": "2020-12-04T13:14:01.038Z",
    "id": "01EKX32ZSAPSCFC97W5Z423QKY",
    "password": "****",
}

Where:

  • access_limit is the number of times that a download can be initiated using this link before it becomes locked if access_limit_enabled is true for this team
  • access_limit_enabled indicates if the link has a limit on the number of times it can be downloaded
  • active indicates if the link can be used to download the package
  • id is the unique identifier for this link record
  • download_secret is the key that must be provided along with id to request a package token to download the file.
  • email the recipients email address - if not provided then this will be the email of the uploader
  • password indicates if a password was attached to the link - if present and the value is "****" that means a password will be required to retrieve a package token and initiate a download. The number of asterisks is constant and does not expose the length of the password itself.

Note

Take note of the id and download_secret as they are required to access the download page. The download_secret is only returned in this response and cannot be retrieved via the API again.

Once the link has been created an email will be sent to the recipient automatically (if provided) and it can be used to access the package on MASV via the following URL: https://get.massive.app/{link_id}?secret={download_secret}

Where:

  • link_id is the link identifier returned when creating the link above
  • download_secret is the secret key returned when creating the link above

Note

The request will need to be repeated for each additional email recipient or direct-download link you wish to create. Only one recipient per link may be specified.