Today we are going to focus on how to transfer and store sensitive files. Why? Every day, BrandOriented, by implementing Effiana, makes life of commercial teams easier from the perspective of their day-to-day duties, including decision support, integrating data in distri-buted environment, and supporting processes within their organisation. Effiana makes a big difference to almost every aspect of teams' work and performance.
However, there are many mechanisms that are not visible to the users, and these mechanisms are responsible for an extremely important area: security. This is the case, for instance, with file encryption. We will discuss it below.
Before we start our preparation to encrypt files, we need to plan the whole process – there are numerous possibilities, both in terms of encryption algorithms as well as methods of processing encrypted files. For the purposes of this text, we will focus on an encryption method that is similar to the one we use in Effian – however, we will show it in a highly simplified way. The examples are intended to give an idea of how the mechanism works – and not to show the actual code. The method assumes that the files arrive at the server in an encrypted form, and that any access to them is limited to selected users, and there is no need for any re-encryption as a result of changes in the list of recipients. The examples will attempt to show a general idea – and there are many ways in which it can be implemented.
So let's start with a simple code that sends files to the server from the user's browser:
function upload() {
const file = document.getElementById("file").files[0];
fetch('/upload', {
method: 'PUT',
body: file
});
}The code above sends the file to the server in the simplest possible way without encryption. However, our goal is to send an encrypted file that will never reach the server in plaintext. So let's add an encryption mechanism to the function that saves the file – and in order to do this, let's use the cryptojs library:
function upload() {
const file = document.getElementById("file").files[0];
saveFile(file);
}
async function saveFile (file) {
const reader = new FileReader();
reader.addEventListener("load", () => {
fetch('/upload', {
method: 'PUT',
body: CryptoJS.AES.encrypt(reader.result, "<pass>")
});
}, false);
if (file) {
reader.readAsText(file);
}
}In this form, the saveFile function will send the encrypted file directly to the server. However, the example above contains a serious error. The encryption key is the password stored in the code. Obviously, various mechanisms can be used here, i.e. sessions or cookies, but this will still not be very secure, as the password with which we encrypt our file will still be the only protection. For this, we will use RSA asymmetric encryption and use the cryptico library, and for the purposes of this article we will put the public key in the code – ultimately, it should be returned from the server, but at this stage let's assume that’s the case.
const publicKeyString = 'MIGeMA0GCSqGSIb3DQEBAQUAA4GMADCBiAKBgGFxQyrqquJh+W/5wtrD3lrEWaCpEDjcHreBGU37v58GB2i3Vl/3VBP1KUqlVmkre/iAXSOKKKnS2k8cqftPZMGf8+r6BJKX7PJkKIEDkxBjReoaIZsibqGEKYF1ZLKmrvXlBO2+EyMvymv8A8QnJtHWdznwyitAM25LKNRnPVWbAgMBAAE=';
function upload() {
const file = document.getElementById("file").files[0];
saveFile(file);
}
async function saveFile (file) {
const reader = new FileReader();
reader.addEventListener("load", () => {
fetch('/upload', {
method: 'PUT',
body: cryptico.encrypt(
reader.result, publicKeyString
).cipher
});
}, false);
if (file) {
reader.readAsText(file);
}
}So we already have a mechanism that encrypts the file for us before it reaches the server; to this end, we use RSA asymmetric encryption, which uses a public key for encryption and a private key for decryption. So it's time to move to the other side and look at what the whole mechanism is going to look like on the server side.
To begin with, we need to generate a key pair, and here an important question pops up:
Should every transmitted file be encrypted with the same key?
The answer is obvious – but only in theoretical terms, because depending on which one is given, it may require additional work and resources. Development teams often try to create solutions with as little effort as possible – this is rarely a good idea when it comes to security. Time and time again in recent years (or even months) we hear about password leaks that, after verification, turn out to be poorly secured (or not secured at all). My approach to the subject of security is always to maximise efforts and resources in order to achieve a satisfactory result, even more so in case of an issue that is as sensitive as the storage of files (which are often confidential). Consequently, now we will focus on maximising the effect of encryption, and a separate key pair will be generated for each file.
To start with, we will improve the javascript code that will retrieve a new private key and a file ID before each file upload to the server – and this ID will help assign the correct keys.
function upload() {
const file = document.getElementById("file").files[0];
fetch('/upload', {method: 'HEAD'}).then(function(response) {
saveFile(
file,
response.headers.get("File-Public-Key"),
response.headers.get("File-Id")
);
});
}
async function saveFile (file, publicKeyString, fileId) {
const reader = new FileReader();
reader.addEventListener("load", () => {
fetch('/upload', {
method: 'PUT',
body: cryptico.encrypt(reader.result, publicKeyString).cipher
});
}, false);
if (file) {
reader.readAsText(file);
}
}The "/upload" [HEAD] method should return two necessary elements in the header which will be stored on the server side of the application (e.g. in a database) before being uploaded. In this situation, the "/upload" [HEAD] method generates a key pair and saves it in the database. When we want to make sure that the file cannot be changed and re-encrypted after encryption, we cannot save the complete key pair and leave only the private key, which will be used to decrypt the file.
For the purposes of this text, we have assumed that only private keys are stored, which are additionally encrypted using the AES algorithm, for which the key is stored in the server's memory (file, configuration record, etc.). This is to protect the private key itself from access by unauthorised individuals who may work on the database. Should they gain access to the database, they would not be able to decrypt the files.
File-Public-Key: MIGeMA0GCSqGSIb3DQEBAQUAA4GMADCBiAKBgGFxQyrqquJh+W/5wtrD3lrEWaCpEDjcHreBGU37v58GB2i3Vl/3VBP1KUqlVmkre/iAXSOKKKnS2k8cqftPZMGf8+r6BJKX7PJkKIEDkxBjReoaIZsibqGEKYF1ZLKmrvXlBO2+EyMvymv8A8QnJtHWdznwyitAM25LKNRnPVWbAgMBAAE=
File-Id: 70ec0290-2768-42f4-af90-e378c524264d
|------------|--------------------|--------------------|
| Id | File_ID | File_Key |
|------------|--------------------|--------------------|After making these changes to the code, there should already be an encrypted file on the server and the private key for decryption in encrypted form stored in the application database. However, this is not the end of the encryption process itself, as the data allowing decryption is still missing. Importantly, at this point no user can decrypt the file. For the functionality to get complete, once the file has been uploaded, we should allocate relevant 'permissions' to the owner of the file (i.e. the person who uploads the file to the server). In other words, we need to encrypt the private key of the file for the owner of the file. In order to do this, we will add a private key entry to the encrypted file, which will be encrypted with the owner's individual key.
The idea itself is not new – a similar mechanism has been used in PGP, but instead of re-encrypting the file to which the keys will be added, separate entries with the users' keys will be added to the encrypted file. This will avoid the need for decryption and re-encryption – both in the context of large files and also in the context of security, as the file will not be decrypted and stored on the server. Our file will look more or less as follows:
-----BEGIN EFFIANA KEY-----
ID:24f1b2296b37c229789611a2293ef498bdde0787c0db3ae5ba216ac51a02c
KEY:/Oc8eQ6SJ52zxlIceORE20ZBqP+u4DhmY5i8rCeTv3GHnktDQBKxfZhqp8oAJKX3hlOvBApt54hZY4FupsxTpyKVbonleDETZcjNENs3Tk4GQqKOd9swADEWyQWC1DTOE+h20pEWldHCp9PKPw5wvg==
-----END EFFIANA KEY——
-----BEGIN EFFIANA MESSAGE-----
uAc6JIvkisAwvHXKjPuYmldoVC1sQWLY7J/Z0Ke1Dqy3yPHeMKzjyzzp8WSukWFCMG54A+9J/gdqDsA7lsScCdooSBNm7lEYuRLiopdzPEt72rqIQqKw2PxTvwCJWgfrDn3DZlFNgCs664OuF3S3mWK6cRoq9lybhCwsPCUGQPwDFHARXxMoEqucbgw5x30+UCNMzEODolWfJv0EgiOb4R55MydPOBVpUT9zRDJW95+mK4xQmjYYtr4kog1C436PdWzpMkNOtzOD49QhHS2y06jONZyD2tG4VMqjD+C9qSqK6tX/p/ZRRzNK9B2yqUALGhnH1wuSKrVNCGIIjXdk/0lCkoSCBEF+J0Nnx0erdjpqiU2AkbDtuRZNKliNIPGhj8p+61Mq4v8OO52PoLwa/rx+RvVlv8k5yJv6p1nHIDB1rnzAKdHz9sKIIY7UYeiX6Z7pGjQk5JEgQRWFnoGAoHAszgzhZe88SPAr+KDNadzxPk+U3+wuLng6w4Wn0ZxmQ=
-----END EFFIANA MESSAGE-----Now the file is complete. It contains all the necessary elements, including an identifier and a user key, which we will use to decrypt the file. The identifier can be an email address, a unique user number, etc. In Effian, we use the user ID, which is stored as a UUID; additionally, it is stored as a hash function using the Keccak algorithm (SHA3).
So what does the process of creating an EFFIANA KEY look like?
At the very beginning, we generate a key pair for the user and for the file, as we assumed at the beginning that each file would be encrypted with its unique keys. Then, for the encrypted file, we retrieve the private key – it is exactly the same one that we use for decryption. Now all that is left is to decrypt the private key using AES and encrypt it with the public key generated for the user. For security reasons, we forget the public key and only store the private key in the database.
Here, we can apply various techniques to secure these keys (including, for instance, additional encryption such as AES using a unique token assigned to the user). There are many options, while the most important thing is that no keys should be stored in the same location (server, drive, etc.) as the files encrypted with them.
PLEASE NOTE: The files encrypted in this way cannot be decrypted without the appropriate key. If the keys are deleted from the database, the files become useless and cannot be recovered.
To prevent this, the public key can be encrypted for each file and sent to an external container. In Effiana, we use a dedicated database for this purpose, which only allows the keys to be added to the table, but they cannot be deleted, edited or retrieved. In addition, every now and then all keys are saved in an encrypted security copy that is distributed to several destinations.
In the case of a database, the command might look as follows:
MySQL: "GRANT INSERT ON table TO ‘username’@'localhost’;"
PostgreSQL: "GRANT INSERT ON table TO username;"In next features, we will look at encrypting large files (>1GB), downloading decrypted files, and enlarging the group of individuals who are authorised to decrypt them.
Originally published on LinkedIn.
