A collection of frequently used fields
implemented as custom Ecto
Types
with best-practice following
validation,
sanitising,
transparent
encryption /
decryption
&
hashing functions
to build Privacy Compliant & Security-focussed Phoenix
Apps much faster! π
We found ourselves repeating code
for commonly used fields on each new Phoenix
project/App ...
We wanted a much easier/faster way of building apps
so we created a collection of pre-defined fields
with built-in validation, sanitising and security.
Fields
makes defining Ecto
Schemas faster
and more precise.
An Elixir package that helps you add popular custom types to your Phoenix/Ecto schemas so you can build apps faster!
@dwyl we are firm believers that personal data (Personally Identifiable Information (PII)) should be encrypted "at rest" i.e. all "user" data should be encrypted before being stored in the database. This project makes hashing, encryption and decryption for secure data storage much easier for everyone.
This package was born out of our research into the best/easiest way to encrypt data in
Phoenix
: dwyl/phoenix-ecto-encryption-example
This module is for people building Elixir/Phoenix apps who want to ship simpler and more maintainable code.
We've attempted to make
Fields
as beginner-friendly as possible.
If you get stuck using it or anything is unclear, please ask for help!
Start using Fields
in your Phoenix App today with these 3 easy steps:
Add the fields
package to your list of dependencies in your mix.exs
file:
def deps do
[
{:fields, "~> 2.10.3"}
]
end
Once you have saved the mix.exs
file,
run mix deps.get
in your terminal to download.
In order to use Encryption and Hashing,
you will need to have environment variables
defined for ENCRYPTION_KEYS
and SECRET_KEY_BASE
respectively.
export ENCRYPTION_KEYS=nMdayQpR0aoasLaq1g94FLba+A+wB44JLko47sVQXMg=
export SECRET_KEY_BASE=GLH2S6EU0eZt+GSEmb5wEtonWO847hsQ9fck0APr4VgXEdp9EKfni2WO61z0DMOF
If you need to create a secure SECRET_KEY_BASE
value, please see:
How to create Phoenix secret_key_base
And for ENCRYPTION_KEYS
, see:
How to create encryption keys
In our case we use a
.env
file to manage our environment variables. See: github.com/dwyl/learn-environment-variables
This allows us to securely manage our secret keys in dev without the risk of accidentally publishing them on Github.
When we deploy our Apps, we use our service provider's built-in key management service to securely store Environment Variables. e.g: Environment Variables on Heroku
Each field can be used in place of an Ecto type when defining your schema.
An example for defining a "user" schema using Fields:
schema "users" do
field :first_name, Fields.Name # Length validated and encrypted
field :email, Fields.EmailEncrypted # Validates email then encrypts
field :address, Fields.AddressEncrypted # Trims address string then encrypts
field :postcode, Fields.PostcodeEncrypted # Validates postcode then encrypts
field :password, Fields.Password # Hash password with argon2 industry standard
timestamps()
end
Each field is defined as an
Ecto type,
with the relevant callbacks.
So when you call Ecto.Changeset.cast/4
in your schema's changeset function,
the field will be correctly validated.
For example, calling cast on the :email
field
will ensure it is a valid format for an email address
RFC 5322.
When you load one of the fields into your database,
the corresponding dump/1
callback will be called,
ensuring it is inserted into the database in the correct format.
In the case of Fields.EmailEncrypted
,
it will encrypt the email address
using a given encryption key
before inserting it.
Likewise, when you load a field from the database,
the load/1
callback will be called,
giving you the data in the format you need.
Fields.EmailEncrypted
will be decrypted back to plaintext.
This all happens 100% transparently to the developer.
It's like magic. But the kind where you can
actually understand how it works!
(if you're curious, read the
code
)
Each Field optionally defines an input_type/0
function.
This will return an atom
representing the Phoenix.HTML.Form
input type to use for the Field.
For example: Fields.DescriptionPlaintextUnlimited.input_type
returns :textarea
which helps us render the correct field in a form.
The fields DescriptionPlaintextUnlimited
and HtmlBody
uses
html_sanitize_ex
to remove scripts and help keep your project safe.
HtmlBody
is able to display basic html elements
whilst DescriptionPlaintextUnlimited
displays text.
Remember to use raw
when rendering
the content of your DescriptionPlaintextUnlimited
and HtmlBody
fields
so that symbols such as & (ampersand) and Html are rendered correctly.
e.g:
<p><%= raw @product.description %></p>
Address
- an address for a physical location. Validated and stored as a (plaintext
)String
.AddressEncrypted
- an address for a customer or user which should be stored encrypted for data protection.DescriptionPlaintextUnlimited
- filters any HTML/JS to avoid security issues. Perfect for blog post comments.Encrypted
- a general purpose encrypted field. converts any type of datato_string
and then encrypts it.EmailEncrypted
- validate and strongly encrypt email address to ensure they are kept private and secure.EmailHash
- when an email needs to be looked up fast without decrypting. Salted and hashed with:sha256
.EmailPlaintext
- when an email address ispublic
there's no advantage to encrypting it. e.g. a customer support email.Hash
- a general-purpose hash field using:sha256
, useful if you need to store the hash of a value. (one way)HtmlBody
- useful for storing HTML data e.g in a CMS.Name
- used for personal names that need to be kept private/secure. Max length 35 characters. AES Encrypted.Password
- passwords hashed usingargon2
.PhoneNumberEncrypted
- a phone number that should be kept private gets validated and encrypted.PhoneNumber
- when a phone number is not sensitive information and can be stored in plaintext.Postcode
- validated postcode stored asplaintext
.PostcodeEncrypted
- validated and encrypted.Url
- validate a URL and store asplaintext
(not encrypted)String
UrlEncrypted
- validate a URL and store as AES encryptedBinary
IpAddressPlaintext
- validate an ipv4 and ipv6 address and store asplaintext
IpAddressHash
- hash for ipv4 or ipv6IpAddressEncrypted
- validate an ipv4 and ipv6 address and store as AES encryptedBinary
Detailed documentation available on HexDocs: hexdocs.pm/fields
mix t
mix c
If there is a field that you need in your app
that is not already in the Fields
package,
please open an issue so we can add it!
github.com/dwyl/fields/issues
If you want an in-depth understanding of how automatic/transparent encryption/decryption works using Ecto Types, see: github.com/dwyl/phoenix-ecto-encryption-example
If you are rusty/new on Binaries in Elixir,
take a look at this post by @blackode:
https://medium.com/blackode/playing-with-elixir-binaries-strings-dd01a40039d5
If you have questions, please open an issue: github.com/dwyl/fields/issues
A recent/good example is: issues/169
EmailEncrypted
and
EmailHash
serve very different purposes.
Briefly:
with
encryption
the output is always different
is meant for safely storing sensitive data
that we want to decrypt later
whereas with
hash
the output is always the same
it cannot be "unhashed" but
can be used to
check a value,
i.e. you can lookup a hashed value in a database.
The best way to understand how these work
is to see it for yourself.
Start an
IEx
session in your terminal:
iex -S mix
You should see output similar to the following:
Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit] [dtrace]
Compiling 23 files (.ex)
Generated fields app
Interactive Elixir (1.12.3) - press Ctrl+C to exit (type h() ENTER for help)
That confirms the fields
module has compiled.
Now that you've initialized IEx
,
issue the following commands:
iex> email = "[email protected]"
"[email protected]"
iex(2)> encrypted = Fields.AES.encrypt(email)
<<48, 48, 48, 49, 20, 6, 117, 239, 107, 251, 80, 156, 109, 46, 6, 75, 119, 89,
72, 163, 156, 243, 60, 6, 17, 166, 130, 239, 93, 222, 65, 186, 185, 78, 77, 2,
80, 194, 241, 31, 28, 24, 155, 172, 208, 185, 142, 64, 65, 127>>
Note: the
Fields.EmailEncrypted
uses theAES.encrypt/1
behind the scenes, that's why we are using it here directly. You could just as easily have written:{:ok, encrypted} = Fields.EmailEncrypted.dump(email)
this is just a shorthand.
That output <<48, 48, 48 ... 64, 65, 127>>
is a
bitstring
which is the sequence of bits in memory.
The encrypted data - usually called
"ciphertext" -
is not human readable, that's a feature.
But if you want to decrypt it back to its human-readable form,
simply run:
iex(3)> decrypted = Fields.AES.decrypt(encrypted)
"[email protected]"
So we know that an encrypted value can be decrypted.
In the case of EmailEncrypted
this is useful
when we want to send someone an email message.
For security/privacy,
we want their sensitive personal data to be stored
encrypted in the Database,
but when we need to decrypt it to send them a message,
it's easy enough.
If you run the Fields.AES.encrypt/1
function
multiple times in your terminal,
you will always see different output:
iex(4)> Fields.AES.encrypt(email)
<<48, 48, 48, 49, 168, 212, 210, 53, 233, 104, 27, 235, 199, 43, 87, 74, 3, 2,
211, 114, 187, 229, 157, 182, 37, 34, 209, 37, 66, 160, 30, 126, 238, 180,
146, 133, 227, 53, 245, 228, 119, 191, 117, 247, 37, 176, 130, 110, ...>>
iex(5)> Fields.AES.encrypt(email)
<<48, 48, 48, 49, 196, 170, 48, 97, 75, 206, 148, 204, 41, 149, 64, 50, 27, 56,
112, 19, 53, 108, 86, 153, 154, 53, 53, 97, 232, 133, 97, 88, 214, 254, 40,
84, 65, 227, 75, 123, 212, 222, 63, 221, 176, 130, 11, 173, ...>>
iex(6)> Fields.AES.encrypt(email)
<<48, 48, 48, 49, 201, 239, 104, 101, 140, 232, 0, 216, 183, 168, 220, 130, 24,
236, 205, 220, 239, 112, 112, 168, 86, 235, 84, 115, 108, 116, 16, 234, 184,
72, 111, 144, 245, 1, 125, 207, 230, 68, 126, 111, 84, 83, 23, 90, ...>>
iex(7)> Fields.AES.encrypt(email)
<<48, 48, 48, 49, 176, 131, 145, 182, 128, 43, 11, 100, 253, 73, 179, 144, 139,
45, 211, 156, 155, 117, 119, 59, 152, 148, 45, 36, 95, 141, 35, 242, 182, 51,
235, 162, 186, 132, 23, 34, 174, 171, 157, 115, 54, 211, 124, 247, ...>>
The first 4 bytes <<48, 48, 48, 49,
are the same
because we are using the same encryption key.
But the rest is always different.
A hash
function
can be used to map data of arbitrary size
to fixed-size values.
i.e. any length of plaintext
will
result in the same length hash
value.
A hash
function is one-way,
it cannot be reversed or "un-hashed".
The hash
value is always the same
for a given string of plaintext.
Try it in IEx
:
iex(1)> email = "[email protected]"
"[email protected]"
iex(2)> Fields.Helpers.hash(:sha256, email)
<<95, 251, 251, 204, 181, 59, 239, 4, 218, 193, 35, 20, 223, 131, 219, 101, 30,
17, 97, 146, 103, 115, 3, 185, 230, 137, 218, 137, 209, 111, 48, 236>>
iex(3)> Fields.Helpers.hash(:sha256, email)
<<95, 251, 251, 204, 181, 59, 239, 4, 218, 193, 35, 20, 223, 131, 219, 101, 30,
17, 97, 146, 103, 115, 3, 185, 230, 137, 218, 137, 209, 111, 48, 236>>
iex(4)> Fields.Helpers.hash(:sha256, email)
<<95, 251, 251, 204, 181, 59, 239, 4, 218, 193, 35, 20, 223, 131, 219, 101, 30,
17, 97, 146, 103, 115, 3, 185, 230, 137, 218, 137, 209, 111, 48, 236>>
The hash value is identical for the given input text
in this case the email address "[email protected]"
.
If you use the Fields.EmailHash.dump/1
function,
you will see the same hash value
(because the same helper function is invoked):
iex(5)> Fields.EmailHash.dump(email)
{:ok,
<<95, 251, 251, 204, 181, 59, 239, 4, 218, 193, 35, 20, 223, 131, 219, 101, 30,
17, 97, 146, 103, 115, 3, 185, 230, 137, 218, 137, 209, 111, 48, 236>>}
iex(6)> Fields.EmailHash.dump(email)
{:ok,
<<95, 251, 251, 204, 181, 59, 239, 4, 218, 193, 35, 20, 223, 131, 219, 101, 30,
17, 97, 146, 103, 115, 3, 185, 230, 137, 218, 137, 209, 111, 48, 236>>}
When the EmailHash
is stored in a database
we can lookup an email address by hashing it
and comparing it to the list.
The best way of visualizing this
is to convert the hash value (bitstring)
to base64
so that it is human-readable:
iex(1)> email = "[email protected]"
"[email protected]"
iex(2)> Fields.Helpers.hash(:sha256, email) |> :base64.encode
"X/v7zLU77wTawSMU34PbZR4RYZJncwO55onaidFvMOw="
iex(3)> Fields.Helpers.hash(:sha256, email) |> :base64.encode
"X/v7zLU77wTawSMU34PbZR4RYZJncwO55onaidFvMOw="
Imagine you have a database table called people
that has just 3 columns: id
, email_hash
and email_encrypted
id |
email_hash |
email_encrypted |
---|---|---|
1 | X/v7zLU77wTawSMU34PbZR4RYZJncwO55onaidFvMOw= | MDAwMc57Y1j0nhwOdw7EvNeUVEfYQoAr7aT6oX |
2 | +zXMhia/Z2I64nul6pqoDZTVM1q2K21Pby6GtPcm9iE= | MDAwMXnS1uwGN/cZRFkQgArm2Sbj9y+hnUJIS7 |
3 | maY4IxoRSOSqm6qyJDrnEN1JQssJRqRGhzwOown4DPU= | MDAwMa4v0FBko++zqfAkfisXOLosQfrDLAdPax |
With this "database" table,
we can now lookup an email address to find out their id
:
iex(4)> Fields.Helpers.hash(:sha256, "[email protected]") |> :base64.encode
"+zXMhia/Z2I64nul6pqoDZTVM1q2K21Pby6GtPcm9iE="
This matches the email_hash
in the second row of our table,
therefore Alice's id
is 2
in the database.