Intro

Standard File is a sync and encryption library for web and native applications. It allows developers to focus on building great user-facing applications, and leaves the syncing, servers, and end-to-end encryption to our framework.

To build a quality application, most app developers today have to implement not only their own front-end clients, but also a backend architecture to handle model storage for their specific schema. With the growing trend of 'experimental' and 'single-use' applications, it becomes impractical to build a new server infrastructure for every application.

Standard File is a reusable client and server system that allows you to deploy a "dumb" backend server that doesn't know or care about your data schema, and allows you to encrypt data on the client-side and sync it with the remote server.

Standard File makes use of progressions in consumer device performance and capacity to enable end-to-end encryption on every platform. One can build any sort of secure and private application with a Standard File server, such as an encrypted notes app like Standard Notes, or any sort of todo or personal data app.

Protocol Specification

Version 0.0.2

(For the 0.0.1 specification, see here)

While most protocol specifications are hard to read and unnecessarily complex, Standard File aims to be a simple system that any developer can understand and implement.

Server Implementations

Ruby Implementation

Go Implementation

Introduction

This document outlines the specifications for the client-server communications of the Standard File system. Any client can communicate with a Standard File server as long as it understands the server's requirements.

Models

The protocol consists of models on the server side and what are known as structures on the client side.

Server Models

  • User
  • Item

Client Structures

  • Arbitrary

An Item model has a content field. The content field stores a JSON encoded object that can be any thing the client needs to operate. In this client-server model, servers are to be treated as dumb and uninformed.

Relationships are handled by the client and not the server, which clients today have no problem handling. This allows for improvements to be made to the data model on the client level, and not on the difficult-to-change server level. It also allows for relationships to be encrypted.

User

A user model has the following properties:

name type description
email String The email of the user.
password String The password for this user. Note that passwords must be manipulated before being sent to the server.
pw_cost String The number of iterations to use for the KDF. See Encryption for more.
pw_salt String Generated by the client during registration. See Encryption for more.

Items

Item models have the following properties:

name type description
uuid String (or uuid for some databases) The unique identifier for this model.
content Text The JSON string encoded structure of the note, encrypted.
content_type String The type of the structure contained in the content field.
enc_item_key Text The locally encrypted encryption key for this item.
deleted Bool Whether the model has been deleted.
created_at Date The date this item was created.
updated_at Date The date this item was modified.

Client Structures

Client structures are stored in the content field of the Item model. A client structure can have any property the client chooses, as well as the following:

name type description
references Array A metadata array of other Item uuids this model is related to and their respective content_type. See sample below.

references array sample:

[
  {uuid: xxxx, content_type: "Tag"},
  {uuid: xxxxx, content_type: "Tag"},
]

REST API

General:

  1. All requests after the initial authentication should be authenticated with a JWT with the Authorization header:

    Authorization: Bearer _insert_JWT_here_
    

Auth

Standard File uses JSON Web Tokens (JWT) for authentication.

POST auth

Registers a user and returns a JWT

Params: email, password, pw_cost, pw_salt

Note: password needs to be processed locally before being sent to the server. See Encryption for more. Never send the user's inputted password to the server.

Responses

200

{"jwt" : "..."}

5xx

{"errors" : []}

PATCH auth

Updates a user's password.

Params: email, password, password_confirmation, current_password

Responses 204

No Content

5xx

{"errors" : []}

POST auth/sign_in

Authenticates a user and returns a JWT.

Note: Passwords needs to be processed locally before being sent to the server. See Encryption for more. Never send the user's inputted password to the server.

Params: email, password

Responses 200

{"token" : "..."}

5xx

{"errors" : []}

GET auth/params

Returns the parameters used for password generation.

Params: email

Responses 200

{"pw_cost" : "...", "pw_salt" : "..."}

5xx

{"errors" : []}

Items

POST items/sync

Saves local changes as well as retrieves remote changes.

Params:

  • items: An array of items
  • sync_token: the sync token returned from the previous sync call. Leave empty if first sync.
  • limit: (optional) the number of results to return. cursor_token is returned if more results are available.

Responses

200

{"retrieved_items" : [], "saved_items" : [], "unsaved_items" : [], "sync_token" : ""}

5xx

{"errors" : []}

Sync Discussion

Deletion:

  • Clients: set deleted equal to true and sync. When receiving an item that is deleted, remove it from the local database immediately.
  • Servers: if syncing an item that is deleted, clear out its content and enc_item_key fields, set deleted to true, and save.

Sync completion:

Upon sync completion, the client should handle each response item as follows:

  • retrieved_items: these items are new or have been modified since last sync and should be merged or created locally.
  • saved_items: saved items are dirty items that were sent to the sync request. These items should not be merged in their entirety upon completion. Instead, only their metadata should be merged. For example, if at Point A the client syncs a Note item that a user is still typing, and at Point B the sync completes, the user could have typed more content in between A and B. Thus, if you merge all content, the user's progress in between A and B will be lost. However, if you merge just the metadata, then this issue does not occur.
  • unsaved: returned if an error occurred saving those items. The only reason this should happen is in the improbable case of a UUID conflict.
  • sync_token: this token should be saved when it is received and sent to subsequent sync requests. This token should also be persisted locally between app sessions. For first time sync, no token should be provided.
  • cursor_token: returned if original request had a limit. Send this token back to the server to retrieve next page of results.

Import/Export

The export file is a JSON file of all the user's items, unencrypted.

Format:

{
  "items": [
    {
      "uuid": "3162fe3a-1b5b-4cf5-b88a-afcb9996b23a",
      "content_type": "Note",
      "content": {
        "references": [
          {
            "uuid": "901751a0-0b85-4636-93a3-682c4779b634",
            "content_type": "Tag"
          }
        ],
        "title": "...",
        "text": "..."
      },
      "created_at": "2016-12-16T17:37:50.000Z"
    },

    {
      "uuid": "023112fe-9066-481e-8a63-f15f27d3f904",
      "content_type": "Tag",
      "content": {
        "references": [
          {
            "uuid": "94cba6b7-6b55-41d6-89a5-e3db8be9fbbf",
            "content_type": "Note"
          }
        ],
        "title": "essays"
      },
      "created_at": "2016-12-16T17:13:20.000Z"
    }
  ]
}

Encryption

Encryption and security should always be top of mind with Standard File.

It is important that there exist a separation of concerns between the server and the client. That is, the client should not trust the server, and vice versa.

Encryption keys are generated by stretching the user's input password using a key derivation function.

The resulting key is split in three — the first third is sent to the server as the user's password, the second third is saved locally as the user's master encryption key, and the last third is used as an authentication key. In this setup, the server is never able to compute the encryption key or the user's original password.

Note: client-server connections must be made securely through SSL/TLS.

name details
pw_cost The number of iterations to be used by the KDF. On native platforms, this should be at least 100,000. However note that non-native clients (web clients not using WebCrypto) will not be able to handle any more than 3,000 iterations.
pw_salt A salt for password derivation. This value is initially created by the client during registration.

Key Generation

Client Instructions

Given a user inputted password uip, the client's job is to generate a password pw to send to the server, a master key mk that the user stores locally to encrypt/decrypt data, and an auth key ak for authenticating server params.

Login Steps

  1. Client makes GET request with user's email to auth/params to retrieve password salt and cost.
  2. Client verifies cost >= minimum cost (3000 for 002, increases every year.)
  3. Client computes pw, mk, and ak using PBKDF2 with SHA512 as the hashing function and output length of 768 bits:

    key = pbkdf2(uip, pw_salt, sha512, 768, pw_cost) // hex encoded
    pw = key.substring(0, key.length/3)
    mk = key.substring(key.length/3, key.length/3)
    ak = key.substring(key.length/3 * 2, key.length/3)
    
  4. Client sends pw to the server as the user's "regular" password and stores mk and ak locally. (mk and ak are never sent to the server).

Registration Steps

  1. Client chooses default for pw_cost (minimum 3000).
  2. Client generates pw_salt:

    nonce = random_string(128) // 128 bit random key
    pw_salt = SHA1:Hexdigest([email, nonce].join(":"))
    
  3. Client computes pw, mk, and ak using step (2) from Authentication Steps.

  4. Client registers with email, pw, pw_cost, pw_salt.

Item Encryption

In general, when encrypting a string, one should use an IV so that two subsequent encryptions of the same content yield different results, and one should authenticate the data as to ascertain its authenticity and lack of tampering.

In Standard File, two strings are encrypted for every item:

  • The item's content.
  • The item's enc_item_key.

Client-side

An item is encrypted using a random key generated for each item.

Encryption:

Encryption version: 002.

Note that when encrypting/decrypting data, keys should be converted to the proper format your platform function supports. It's best to convert keys to binary data before running through any encryption/hashing algorithm.

For every item:

  1. Generate a random 512 bit key item_key (in hex format).
  2. Split item_key in half; set item encryption key item_ek = first_half and item authentication key item_ak = second_half.
  3. Encrypt content using item_ek and item_ak following the instructions "Encrypting a string using the 002 scheme" below and send to server as content.
  4. Encrypt item_key using the global mk and global ak following the instructions "Encrypting a string using the 002 scheme" below and send to server as enc_item_key.

Decryption:

Check the first 3 characters of the content string. This will be the encryption version.

  • If it is equal to "001", which is a legacy scheme, decrypt according to the 001 instructions found here.

  • If it is equal to "002", which is the current scheme, decrypt as follows:

    1. Decrypt enc_item_key using the global mk and global ak according to the "Decrypting a string using the 002 scheme" instructions below to get item_key.
    2. Split item_key in half; set encryption key item_ek = first_half and authentication key item_ak = second_half.
    3. Decrypt content using item_ek and item_ak according to the "Decrypting a string using the 002 scheme" instructions below.

Encrypting a string using the 002 scheme:

Given a string_to_encrypt, an encryption_key, and an auth_key:

  1. Generate a random 128 bit string called IV.
  2. Encrypt string_to_encrypt using AES-CBC-256:Base64, encryption_key, and IV:

    ciphertext = AES-Encrypt(string_to_encrypt, encryption_key, IV)
    
  3. Generate string_to_auth by combining the encryption version (002), the item's UUID, the IV, and the ciphertext using the colon ":" character:

    string_to_auth = ["002", uuid, IV, ciphertext].join(":")
    
  4. Compute auth_hash = HMAC-SHA256:Hex(string_to_auth, auth_key).

  5. Generate the final result by combining the five components into a : separated string:
    result = ["002", auth_hash, uuid, IV, ciphertext].join(":")
    

Decrypting a string using the 002 scheme:

Given a string_to_decrypt, an encryption_key, and an auth_key:

  1. Split the string into its constituent parts: components = string_to_decrypt.split(":").
  2. Assign local variables:

    version = components[0]
    auth_hash = components[1]
    uuid = components[2]
    IV = components[3]
    ciphertext = components[4]
    
  3. Ensure that uuid === item.uuid. If not, abort decryption.

  4. Generate string_to_auth = [version, uuid, IV, ciphertext].join(":").
  5. Compute local_auth_hash = HMAC-SHA256(string_to_auth, auth_key). Compare local_auth_hash to auth_hash. If they are not the same, skip decrypting this item, as this indicates that the string has been tampered with.
  6. Decrypt ciphertext to get final result: result = AES-Decrypt(ciphertext, encryption_key, IV).

Server-side

For every received item:

  1. (Optional but recommended) Encrypt content using server known key and store. Decrypt before sending back to client.

Next Steps

Check out the client development guide for a practical guide to developing an application on top of Standard File.

See Standard Notes Developer Resources.

Join the Slack group to discuss implementation details and ask any questions you may have.

You can also email hello@standardnotes.org.

Follow @standardnotes for updates and announcements.

results matching ""

    No results matching ""