-
Notifications
You must be signed in to change notification settings - Fork 8
Complete Driver Implementation
This page includes instructions for implementing complete FMKe drivers, which implement the complete application interface and work closely with database APIs. They can provide greater performance results than simple drivers.
- Some Erlang/OTP experience or willingness to learn some basic constructs
- Availability of an Erlang client library for the database of choice. Fortunately, Erlang has a large amount of database client library implementations widely available, however, not all of them are compatible with the latest Erlang versions. To qualify for eligibility, the library needs to be compatible with Erlang 20 or newer.
- Not only do you need an Erlang client library for your database of choice, you need to become familiar with it. It is possible that several options are available to achieve a certain goal, and rigorous testing may be required in order to determine which approach yields the best results.
You also need to install a version of Erlang and rebar3 that is compatible with the currently supported versions. Check the main repository's README.md file in order to see which versions are supported.
If your goal is to contribute to FMKe (we'd like that!), fork the repository and add a new file in the src folder - FMKe drivers follow a specific notation: fmke_driver_opt_DATABASE.erl, where DATABASE is the name of the database. It is possible that your driver implements logic for multiple data models, but as this is generally not the case, you may also wish to suffix a data model reference. If so, the final file name would become fmke_driver_opt_DATABASE_DATAMODEL.erl.
If we were implementing a nested data model for the Riak database, we could name the file fmke_driver_opt_riak_nested.erl.
In the first few lines of code we add the following:
-module(fmke_driver_opt_riak_nested).
-behaviour(fmke_gen_driver).
-behaviour(gen_server).If we try to compile the project now with the rebar3 compile command, we will get a list of errors corresponding to several functions that we still need to implement:
===> Compiling fmke
===> Compiling src/fmke_driver_opt_riak_nested.erl failed
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function create_facility/4 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function create_patient/3 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function create_pharmacy/3 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function create_prescription/6 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function create_staff/4 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function get_facility_by_id/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function get_patient_by_id/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function get_pharmacy_by_id/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function get_pharmacy_prescriptions/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function get_prescription_by_id/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function get_processed_pharmacy_prescriptions/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function get_staff_by_id/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function get_staff_prescriptions/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function process_prescription/2 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function start/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function stop/1 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function update_facility_details/4 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function update_patient_details/3 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function update_pharmacy_details/3 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function update_prescription_medication/3 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:6: undefined callback function update_staff_details/4 (behaviour 'fmke_gen_driver')
src/fmke_driver_opt_riak_nested.erl:7: undefined callback function handle_call/3 (behaviour 'gen_server')
src/fmke_driver_opt_riak_nested.erl:7: undefined callback function handle_cast/2 (behaviour 'gen_server')
src/fmke_driver_opt_riak_nested.erl:7: undefined callback function init/1 (behaviour 'gen_server')
Now we know which functions we need to implement. If you aren't familiar with the gen_server behaviour, you can look at the official documentation or you are welcome to ask for help. It is a basic OTP construct and it is used in every FMKe driver, so you need to incorporate it in your code.
The driver is controlled via the start/1 and stop/1 callbacks, so perhaps that is where you should start:
-define(SERVER, ?MODULE).
start(Args) ->
%% just start the gen_server from here and pass it the arguments.
gen_server:start_link({local, ?SERVER}, ?MODULE, Args, []).
stop(_) ->
%% stop the gen_server, log anything useful, etc.
gen_server:stop(?SERVER).If you've written your fair share of gen_server implementations, you might be able to guess where this is going. Every application level operation will in some way invoke a gen_server call. For convenience, here is a typical result of transforming FMKe application operations into gen_server calls:
create_patient(Id, Name, Address) ->
gen_server:call(?SERVER, {create, patient, [Id, Name, Address]}).
create_pharmacy(Id, Name, Address) ->
gen_server:call(?SERVER, {create, pharmacy, [Id, Name, Address]}).
create_facility(Id, Name, Address, Type) ->
gen_server:call(?SERVER, {create, facility, [Id, Name, Address, Type]}).
create_staff(Id, Name, Address, Speciality) ->
gen_server:call(?SERVER, {create, staff, [Id, Name, Address, Speciality]}).
create_prescription(PrescriptionId, PatientId, PrescriberId, PharmacyId, DatePrescribed, Drugs) ->
gen_server:call(?SERVER,
{create, prescription, [PrescriptionId, PatientId, PrescriberId, PharmacyId, DatePrescribed, Drugs]}
).
get_facility_by_id(Id) ->
gen_server:call(?SERVER, {read, facility, Id}).
get_patient_by_id(Id) ->
gen_server:call(?SERVER, {read, patient, Id}).
get_pharmacy_by_id(Id) ->
gen_server:call(?SERVER, {read, pharmacy, Id}).
get_processed_pharmacy_prescriptions(Id) ->
gen_server:call(?SERVER, {read, pharmacy, Id, processed_prescriptions}).
get_pharmacy_prescriptions(Id) ->
gen_server:call(?SERVER, {read, pharmacy, Id, prescriptions}).
get_prescription_by_id(Id) ->
gen_server:call(?SERVER, {read, prescription, Id}).
get_prescription_medication(Id) ->
gen_server:call(?SERVER, {read, prescription, Id, [drugs]}).
get_staff_by_id(Id) ->
gen_server:call(?SERVER, {read, staff, Id}).
get_staff_prescriptions(Id) ->
gen_server:call(?SERVER, {read, staff, Id, prescriptions}).
process_prescription(Id, DateProcessed) ->
gen_server:call(?SERVER, {update, prescription, Id, {date_processed, DateProcessed}}).
update_patient_details(Id, Name, Address) ->
gen_server:call(?SERVER, {update, patient, [Id, Name, Address]}).
update_pharmacy_details(Id, Name, Address) ->
gen_server:call(?SERVER, {update, pharmacy, [Id, Name, Address]}).
update_facility_details(Id, Name, Address, Type) ->
gen_server:call(?SERVER, {update, facility, [Id, Name, Address, Type]}).
update_staff_details(Id, Name, Address, Speciality) ->
gen_server:call(?SERVER, {update, staff, [Id, Name, Address, Speciality]}).
update_prescription_medication(Id, Operation, Drugs) ->
gen_server:call(?SERVER, {update, prescription, Id, {drugs, Operation, Drugs}}).These callbacks to gen_server basically imply that a single process will handle all application requests in the order they are received. What remains are the init/1, handle_cast/2 and handle_call/3 callbacks. We provide a possible implementation with our runnning riak_nested driver example:
init([DataModel]) ->
%% start the riak client library
{ok, _Started} = application:ensure_all_started(riak_client),
GenServerState = {},
{ok, GenServerState}.
handle_cast(_Msg, State) ->
%% we don't support asynchronous operations, so we can ignore any received message and keep the state.
{noreply, State}.
handle_call({read, facility, Id}, _From, State) ->
Pid = fmke_db_conn_manager:checkout(),
Bucket = {<<"maps">>, <<"facilities">>},
Key = get_key(facility, Id),
Result = riakc_pb_socket:fetch_type(Pid, Bucket, Key),
fmke_db_conn_manager:checkin(Pid),
Reply = case Result of
{error, {notfound, _Type}} ->
{error, not_found};
{ok, Map} ->
unmarshall(facility, Map)
end,
{reply, Reply, State};
handle_call(_, _, State) ->
%% address all the remaining cases here.
%% It is possible that some code is common between different entities, so you can extract
%% the common parts to separate functions and reduce code redundancy. Also check other
%% drivers for ''inspiration'' :)
%% TODO: implement
{noreply, State}.
get_key(Entity, Id) ->
list_to_binary(lists:flatten(io_lib:format("~p_~p", [Entity, Id]))).In the interest of time, we cannot provide a complete implementation of all the features, but after implementing the create and read callbacks for the same entity you will be able to verify if your code is running as expected once you perform the required wiring that is described in the following sections.
Looking at the previous code, we reference fmke_db_conn_manager which we had not yet addressed. FMKe uses poolboy, a popular Erlang worker pool, to store connections to the database you are connecting to. This is optional, and we will be looking into how to set that up now.
The reason FMKe exposes a module for connection management is to rotate between the connection pools in a round-robin fashion. This allows FMKe to create a connection pool to each database server and distribute load over the database cluster uniformly.
FMKe provides a connection management mechanism that is able to operate with most client libraries that expose a single connection callback. This is considered a more advanced parameter to configure manually, so we advise you try to use the
Your client library must have a start_link(Hostname :: list(), Port :: non_neg_integer()). Client libraries for AntidoteDB and Riak expose this type of interface, but your particular database may not. You can then choose to either fork the client library and add that feature yourself, or create a new wrapper module that does that and send it along with the rest of your code when you submit a pull request to FMKe. While we would prefer the first approach, the second one is acceptable.
Go into src/fmke_setup_sup.erl and add an additional clause for your database that indicates that the connection manager is required when booting up FMKe:
-spec requires_conn_manager(Database::atom(), DataModel::atom(), Optimized::atom()) -> true | false.
requires_conn_manager(antidote, _, _) -> true;
requires_conn_manager(riak, _, _) -> true;
requires_conn_manager(_, _, _) -> false.Riak is already supported, and since there is already a clause for riak, in our running example we don't need to add anything in.
As seen from the point, the connection manager's policy is opt-in. If you've chosen not to use FMKe's connection manager (because your client library manages it for you, or for your optimisation purposes), you must initialize it properly inside your driver's init/1 function.
ETS is an Erlang key-value storage system that is built in to the Erlang RunTime System (ERTS). You can request that an instance of it (a table called "fmke") is created, and you can use it, for instance, to add a caching layer to your driver. In order to request that the table be created for you, change the following code:
-spec requires_ets_table(Database::atom(), DataModel::atom(), Optimized::atom()) -> true | false.
requires_ets_table(ets, _, _) -> true;
requires_ets_table(_, _, _) -> false.In the requires_ets_table/3 function, add a clause to your database if you want to have a dedicated ETS table. Of course, since you are implementing a complete driver you can also create this table manually instead.
As you implement all the required functions, you will find that if you have performed all of these steps but have not completed all different handle_call/3 implementations, you will be able to run FMKe (but the operations you did not implement will not work). Look for Docker images that successfully start up the database in a distributed setting. there are some docker images that contain a single instance that is tricked or configured into operating like it is working in a distributed environment. Start the Docker image, edit config/fmke.config and see if rebar3 shell works for you. If you get a working Erlang shell, test out all of the different operations you've implemented and see if they work as expected.
There are several checks you can perform after writing your code that will signal possible implementation errrors:
Use rebar3 dialyzer to run the optional Erlang type checker against the module you've written. It takes a few minutes the first time it runs, but it is crucial in determining whether you've respected FMKe's interfaces, which
are important to ensure uniform reply values and correct JSON marshalling.
This is another Erlang tool that checks if functions you've called within your driver are all implemented. You can run an xref sweep using rebar3 xref.
There are several unit tests scattered across FMKe's modules. It is unlikely that your changes broke one of these tests, but you can verify by running rebar3 eunit.
As stated previously, if you find an issue or face difficulties using any of these tools, open an issue and we will do our best to help you get things working.
FMKe uses more intricate integration tests with all supported databases to prevent "it works on my machine" situations. These take the form of two critical tests suites, one that tests the HTTP REST API, and the underlying marshalling and unmarshalling process behind it, and another which consists on performing multiple different operations that test a wide range of possible interactions with the application server. These suites must be passing if the driver you have written is to be accepted.
To use the driver in a somewhat realistic scenario, you will need to find a working Docker image for your database. FMKe does not use mocking to simulate database behaviour. Once you found a compatible Docker image, glance over the code of test/fmke_test_setup.erl. Most of it relates to starting up the docker images for the testing suites that are in the same folder. The main function that is called for this purpose is start_db/1:
start_db(antidote, Port) ->
start_antidote(Port);
start_db(ets, _Port) ->
ok; %% ets doesn't need to be started
start_db(redis, Port) ->
start_redis(Port);
start_db(riak, Port) ->
start_riak(Port).Add a clause for your database using the others as inspiration. Please use lightweight images that spawn as few instances as possible, as tests running in a Continuous Integration (CI) environment may be resource restricted. Once this is done, you can use the instructions in the testing drivers page in order to see if the two critical suites pass with the driver you've created.
If these tests pass, then you've successfully implemented a complete FMKe driver! Now you can use it to compare the performance of your database against others and see how they stack up against the code that you've written. It is possible that through investigating potential performance bottlenecks in your driver and performance profiling that you iterate through several versions, eventually coming up with multiple performance optimisations.
- Benchmarking
- Implementing Drivers
- Testing
- Documentation