- HeteroVector<std::size_t A = 0>: This is an actual heterogenous vector that would contain data. This will result in a "standard" data frame
diff --git a/docs/HTML/io_format.html b/docs/HTML/io_format.html
index 7f6807b5..09479fe3 100644
--- a/docs/HTML/io_format.html
+++ b/docs/HTML/io_format.html
@@ -47,7 +47,7 @@
csv2 = 2, // Regular csv format (similar to Pandas)
json = 3,
hdf5 = 4, // Not Implemented
- binary = 5, // Not Implemented
+ binary = 5,
};
diff --git a/docs/HTML/read.html b/docs/HTML/read.html
index e7511a04..b56e1534 100644
--- a/docs/HTML/read.html
+++ b/docs/HTML/read.html
@@ -95,7 +95,9 @@
- Column "INDEX" must be the first column, if it exists
- Fields in column dictionaries must be in N (name), T (type), D (data) order
-
+ -----------------------------------------------
+ Binary format is a proprietary format, that is optimized for compressing algorithms. It also takes care of different endianness. The file is always written with the same endianness as the writing host. But it will be adjusted accordingly when reading it from a different host with a different endianness.
+
-----------------------------------------------
In all formats the following data types are supported:
@@ -117,8 +119,8 @@
string -- char *
bool -- bool
DateTime -- DateTime data in format of
- <Epoch seconds>.<nanoseconds>
- (1516179600.874123908)
+ <Epoch seconds>.<nanoseconds>
+ (1516179600.874123908)
In case of io_format::csv2 and io_format::csv the following additional types are also supported:
diff --git a/docs/HTML/write.html b/docs/HTML/write.html
index 0820eb15..b0b00c4e 100644
--- a/docs/HTML/write.html
+++ b/docs/HTML/write.html
@@ -41,7 +41,7 @@
- |
+ |
template<typename S, typename ... Ts>
bool
@@ -52,39 +52,41 @@
long max_recs = std::numeric_limits::max()) const;
|
-
- It outputs the content of DataFrame into the stream o. Currently 3 formats (i.e. csv, csv2, json) are supported specified by the iof parameter.
+ |
+ It outputs the content of DataFrame into the stream o. Currently 4 formats (i.e. csv, csv2, json, binary) are supported specified by the iof parameter.
The CSV file format is written:
- INDEX:<Number of data points>:<Comma delimited list of values>
- <Column1 name>:<Number of data points>:<Column1 type>:<Comma delimited list of values>
- <Column2 name>:<Number of data points>:<Column2 type>:<Comma delimited list of values>
- .
- .
- .
+INDEX:<Number of data points>:<Comma delimited list of values>
+<Col1 name>:<Number of data points>:<Col1 type>:<Comma delimited list of values>
+<Col2 name>:<Number of data points>:<Col2 type>:<Comma delimited list of values>
+ .
+ .
+ .
All empty lines or lines starting with # will be skipped. For examples, see files in test directory
The CSV2 file format must be (this is similar to Pandas csv format):
- INDEX:<Number of data points>:<Index type>:,<Column1 name>:<Number of data points>:<Column1 type>,<Column2 name>:<Number of data points>:<Column2 type>, . . .
- Comma delimited rows of values
- .
- .
- .
+INDEX:<Number of data points>:<Index type>:,<Column1 name>:
+<Number of data points>:<Column1 type>,<Column2 name>:
+<Number of data points>:<Column2 type>, . . .
+Comma delimited rows of values
+ .
+ .
+ .
All empty lines or lines starting with # will be skipped. For examples, see IBM and FORD files in test directory
The JSON file format looks like this:
- {
- "INDEX":{"N":3,"T":"ulong","D":[123450,123451,123452]},
- "col_3":{"N":3,"T":"double","D":[15.2,16.34,17.764]},
- "col_4":{"N":3,"T":"int","D":[22,23,24]},
- "col_str":{"N":3,"T":"string","D":["11","22","33"]},
- "col_2":{"N":3,"T":"double","D":[8,9.001,10]},
- "col_1":{"N":3,"T":"double","D":[1,2,3.456]}
- }
+{
+ "INDEX":{"N":3,"T":"ulong","D":[123450,123451,123452]},
+ "col_3":{"N":3,"T":"double","D":[15.2,16.34,17.764]},
+ "col_4":{"N":3,"T":"int","D":[22,23,24]},
+ "col_str":{"N":3,"T":"string","D":["11","22","33"]},
+ "col_2":{"N":3,"T":"double","D":[8,9.001,10]},
+ "col_1":{"N":3,"T":"double","D":[1,2,3.456]}
+}
Please note DataFrame json does not follow json spec 100%. In json, there is no particular order in dictionary fields. But in DataFrame json:
@@ -93,66 +95,75 @@
+
+ Binary format is a proprietary format, that is optimized for compressing algorithms. It also takes care of different endianness. The file is always written with the same endianness as the writing host. But it will be adjusted accordingly when reading it from a different host with a different endianness.
+
In all formats the following data types are supported:
- float -- float
- double -- double
- longdouble -- long double
- short -- short int
- ushort -- unsigned short int
- int -- int
- uint -- unsigned int
- long -- long int
- longlong -- long long int
- ulong -- unsigned long int
- ulonglong -- unsigned long long int
- char -- char
- uchar -- unsigned char
- string -- std::string
- string -- const char *
- string -- char *
- bool -- bool
- DateTime -- DateTime data in format of <Epoch seconds>.<nanoseconds> (1516179600.874123908)
+float -- float
+double -- double
+longdouble -- long double
+short -- short int
+ushort -- unsigned short int
+int -- int
+uint -- unsigned int
+long -- long int
+longlong -- long long int
+ulong -- unsigned long int
+ulonglong -- unsigned long long int
+char -- char
+uchar -- unsigned char
+string -- std::string
+string -- const char *
+string -- char *
+bool -- bool
+DateTime -- DateTime data in format of
+ <Epoch seconds>.<nanoseconds>
+ (1516179600.874123908)
In case of io_format::csv2 and io_format::csv the following additional types are also supported:
- dbl_vec -- A vector of double precision values, The vector is printed as "s[d1|d2|...]"
- where s is the size of the vector and d's are the double values.
- str_vec -- A vector of std::string values, The vector is printed as "s[str1|str2|...]"
- where s is the size of the vector and str's are the strings.
- dbl_set -- A set of double precision values, The set is printed as "s[d1|d2|...]"
- where s is the size of the set and d's are the double values.
- str_set -- A set of std::string values, The set is printed as "s[str1|str2|...]"
- where s is the size of the set and str's are the strings.
- str_dbl_map -- A map of string keys to double precision values, The map is printed as "s{k1:v1|k2:v2|...}"
- where s is the size of the map and k's and v's are keys and values.
- str_dbl_unomap -- An unordered map of string keys to double precision values, The map is printed as "s{k1:v1|k2:v2|...}"
- where s is the size of the map and k's and v's are keys and values.
+dbl_vec -- A vector of double precision values, The vector is printed
+ as "s[d1|d2|...]" where s is the size of the vector and d's
+ are the double values.
+str_vec -- A vector of std::string values, The vector is printed as
+ "s[str1|str2|...]" where s is the size of the vector and
+ str's are the strings.
+dbl_set -- A set of double precision values, The set is printed as
+ "s[d1|d2|...]" where s is the size of the set and d's
+ are the double values.
+str_set -- A set of std::string values, The set is printed as
+ "s[str1|str2|...]" where s is the size of the set and
+ str's are the strings.
+str_dbl_map -- A map of string keys to double precision values, The map is
+ printed as "s{k1:v1|k2:v2|...}" where s is the size of
+ the map and k's and v's are keys and values.
+str_dbl_unomap -- An unordered map of string keys to double precision values,
+ The map is printed as "s{k1:v1|k2:v2|...}" where s is the
+ size of the map and k's and v's are keys and values.
In case of io_format::csv2 the following additional types are also supported:
- DateTimeAME -- DateTime string printed in American style (MM/DD/YYYY HH:MM:SS.mmm)
- DateTimeEUR -- DateTime string printed in European style (YYYY/MM/DD HH:MM:SS.mmm)
- DateTimeISO -- DateTime string printed in ISO style (YYYY-MM-DD HH:MM:SS.mmm)
+DateTimeAME -- American style (MM/DD/YYYY HH:MM:SS.mmm)
+DateTimeEUR -- European style (YYYY/MM/DD HH:MM:SS.mmm)
+DateTimeISO -- ISO style (YYYY-MM-DD HH:MM:SS.mmm)
|
-
-
- S: Output stream type
- Ts: The list of types for all columns. A type should be specified only once
- o: Reference to an streamable object (e.g. cout, file, ...)
- iof: Specifies the I/O format. The default is CSV
- precision: Specifies the precision for floating point numbers
- columns_only: If true, the index columns is not written into the stream
- max_recs: Max number of rows to write. If it is positive, it will write max_recs from the beginning of DataFrame. If it is negative, it will write max_recs from the end of DataFrame
-
+ |
+ S: Output stream type
+ Ts: The list of types for all columns. A type should be specified only once
+ o: Reference to an streamable object (e.g. cout, file, ...)
+ iof: Specifies the I/O format. The default is CSV
+ precision: Specifies the precision for floating point numbers
+ columns_only: If true, the index columns is not written into the stream
+ max_recs: Max number of rows to write. If it is positive, it will write max_recs from the beginning of DataFrame. If it is negative, it will write max_recs from the end of DataFrame
|
- |
+ |
template<typename ... Ts>
std::future<bool>
@@ -163,7 +174,7 @@
long max_recs = std::numeric_limits::max()) const;
|
-
+ |
Same as write() above, but it takes a file name
NOTE:: This version of write() can be substantially faster, especially for larger files, than if you open the file yourself and use the write() version above.
|
@@ -172,7 +183,7 @@
- |
+ |
template<typename S, typename ... Ts>
std::future<bool>
@@ -191,7 +202,7 @@
|
- |
+ |
template<typename ... Ts>
std::future<bool>
@@ -210,14 +221,14 @@
|
- |
+ |
template<typename ... Ts>
std::string
to_string(std::streamsize precision = 12) const;
|
-
+ |
This is a convenient function (simple implementation) to convert a DataFrame into a string that could be restored later by calling from_string(). It utilizes the write() member function of DataFrame.
These functions could be used to transmit a DataFrame from one place to another or store a DataFrame in databases, caches, ...
@@ -229,7 +240,7 @@
|
-
+ |
Ts: The list of types for all columns. A type should be specified only once
precision: Specifies the precision for floating point numbers
|
diff --git a/include/DataFrame/Internals/DataFrame_functors.h b/include/DataFrame/Internals/DataFrame_functors.h
index 4552fc40..a6f4bce3 100644
--- a/include/DataFrame/Internals/DataFrame_functors.h
+++ b/include/DataFrame/Internals/DataFrame_functors.h
@@ -228,6 +228,25 @@ struct print_csv_functor_ : DataVec::template visitor_base {
void operator() (const T &vec);
};
+// ----------------------------------------------------------------------------
+
+template
+struct print_binary_functor_ : DataVec::template visitor_base {
+
+ inline print_binary_functor_ (const char *n,
+ std::ostream &o,
+ long sr,
+ long er)
+ : name(n), os(o), start_row(sr), end_row(er) { }
+
+ const char *name;
+ std::ostream &os;
+ const long start_row;
+ const long end_row;
+
+ template
+ void operator() (const T &vec);
+};
// ----------------------------------------------------------------------------
diff --git a/include/DataFrame/Internals/DataFrame_misc.tcc b/include/DataFrame/Internals/DataFrame_misc.tcc
index 0dd1f27c..e326ff8c 100644
--- a/include/DataFrame/Internals/DataFrame_misc.tcc
+++ b/include/DataFrame/Internals/DataFrame_misc.tcc
@@ -197,16 +197,15 @@ DataFrame::print_csv_functor_::operator() (const T &vec) {
using VecType = typename std::remove_reference::type;
using ValueType = typename VecType::value_type;
- _write_csv_df_header_(os, name, vec.size()) << ':';
+ _write_csv_df_header_(os, name, vec.size())
+ << ':';
const long vec_size = vec.size();
const long sr = std::min(start_row, vec_size);
const long er = std::min(end_row, vec_size);
- if (vec_size > 0) {
- for (long i = sr; i < er; ++i)
- _write_csv_df_index_(os, vec[i]) << ',';
- }
+ for (long i = sr; i < er; ++i)
+ _write_csv_df_index_(os, vec[i]) << ',';
os << '\n';
return;
@@ -214,6 +213,31 @@ DataFrame::print_csv_functor_::operator() (const T &vec) {
// ----------------------------------------------------------------------------
+template
+template
+template
+void
+DataFrame::print_binary_functor_::operator() (const T &vec) {
+
+ using VecType = typename std::remove_reference::type;
+ using ValueType = typename VecType::value_type;
+
+ char col_name[64];
+
+ std::strncpy(col_name, name, sizeof(col_name));
+ os.write(col_name, sizeof(col_name));
+ if constexpr (std::is_same_v)
+ _write_binary_string_(os, vec);
+ else if constexpr (std::is_same_v)
+ _write_binary_datetime_(os, vec);
+ else
+ _write_binary_data_(os, vec);
+
+ return;
+}
+
+// ----------------------------------------------------------------------------
+
template
template
template
@@ -696,7 +720,7 @@ operator() (T &vec) {
using VecType = typename std::remove_reference::type;
using ValueType = typename VecType::value_type;
- using ViewType = typename DF::template ColumnVecType;
+ using ViewType = typename DF::template ColumnVecType;
ViewType new_col;
const size_type vec_size = vec.size();
@@ -737,12 +761,12 @@ operator() (T &vec) const {
if (sel_indices[i] < vec_s) {
if constexpr (std::is_base_of, H>::value)
vec.erase(vec.begin() + (sel_indices[i] - del_count++));
- else
+ else
vec.erase(sel_indices[i] - del_count++);
- }
+ }
else
break;
- }
+ }
}
// ----------------------------------------------------------------------------
@@ -772,7 +796,7 @@ random_load_data_functor_::operator() (const T &vec) {
const size_type vec_s = vec.size();
const size_type n_rows = rand_indices.size();
- typename DF::template ColumnVecType new_vec;
+ typename DF::template ColumnVecType new_vec;
size_type prev_value { 0 };
new_vec.reserve(n_rows);
diff --git a/include/DataFrame/Internals/DataFrame_private_decl.h b/include/DataFrame/Internals/DataFrame_private_decl.h
index bafc83cf..637770c0 100644
--- a/include/DataFrame/Internals/DataFrame_private_decl.h
+++ b/include/DataFrame/Internals/DataFrame_private_decl.h
@@ -61,6 +61,7 @@ using JoinSortingPair = std::pair;
// ----------------------------------------------------------------------------
void read_json_(std::istream &file, bool columns_only);
+void read_binary_(std::istream &file);
void read_csv_(std::istream &file, bool columns_only);
void read_csv2_(std::istream &file,
bool columns_only,
@@ -463,7 +464,7 @@ void remove_data_by_sel_common_(const StlVecType &col_indices) {
std::is_base_of, H>::value)
this->indices_.erase(this->indices_.begin() +
(col_indices[i] - del_count++));
- else
+ else
this->indices_.erase(col_indices[i] - del_count++);
}
};
@@ -490,7 +491,7 @@ void remove_data_by_sel_common_(const StlVecType &col_indices) {
if constexpr (std::is_base_of, H>::value)
indices_.erase(indices_.begin() +
(col_indices[i] - del_count++));
- else
+ else
indices_.erase(col_indices[i] - del_count++);
}
}
@@ -933,7 +934,7 @@ col_vector_push_back_cont_func_(V &vec,
value.reserve(2048);
while (file.get(c)) [[likely]] {
- if (c == '\n') break;
+ if (c == '\n') [[unlikely]] break;
file.unget();
value.clear();
_get_token_from_file_(file, ',', value, '\0');
@@ -990,6 +991,12 @@ struct ColVectorPushBack_, Dummy> {
value.clear();
_get_token_from_file_(file, ',', value,
file_type == io_format::json ? ']' : '\0');
+
+ if (file_type == io_format::json) { // Get rid of "'s
+ value.pop_back();
+ value.erase(value.begin());
+ }
+
vec.push_back(value);
}
}
@@ -1039,7 +1046,7 @@ inline static void
json_str_col_vector_push_back_(StlVecType &vec,
std::istream &file) {
- char value[1024];
+ char value[2048];
char c = 0;
while (file.get(c))
@@ -1060,11 +1067,10 @@ json_str_col_vector_push_back_(StlVecType &vec,
throw DataFrameError(
"json_str_col_vector_push_back_(): ERROR: Expected '\"' (0)");
- while (file.get(c))
- if (c == '"')
- break;
- else
- value[count++] = c;
+ while (file.get(c)) {
+ if (c == '"') break;
+ else value[count++] = c;
+ }
if (c != '"')
throw DataFrameError(
"DataFrame::read_json_(): ERROR: Expected '\"' (1)");
diff --git a/include/DataFrame/Internals/DataFrame_read.tcc b/include/DataFrame/Internals/DataFrame_read.tcc
index 4140f253..2a0d02b0 100644
--- a/include/DataFrame/Internals/DataFrame_read.tcc
+++ b/include/DataFrame/Internals/DataFrame_read.tcc
@@ -28,9 +28,12 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include
+#include
#include
#include
+#include
+
// ----------------------------------------------------------------------------
namespace hmdf
@@ -1270,6 +1273,200 @@ read_csv2_(std::istream &stream,
// ----------------------------------------------------------------------------
+template
+void DataFrame::read_binary_(std::istream &stream) {
+
+ endians ed;
+
+ stream.read(reinterpret_cast(&ed), sizeof(ed));
+
+ const bool needs_flipping = ed != get_system_endian();
+ uint16_t col_num { 0 };
+
+ stream.read(reinterpret_cast(&col_num), sizeof(col_num));
+ if (needs_flipping)
+ col_num = SwapBytes { }(col_num);
+
+ char col_name[64];
+
+ std::memset(col_name, 0, sizeof(col_name));
+ stream.read(col_name, sizeof(col_name));
+ if (std::strcmp(col_name, DF_INDEX_COL_NAME)) {
+ String1K err;
+
+ err.printf("read_binary_(): ERROR: Expecting name '%s'",
+ DF_INDEX_COL_NAME);
+ throw DataFrameError(err.c_str());
+ }
+
+ char col_type[32];
+
+ std::memset(col_type, 0, sizeof(col_type));
+ stream.read(col_type, sizeof(col_type));
+
+ IndexVecType idx_vec;
+
+ if constexpr (std::is_same_v)
+ _read_binary_string_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_datetime_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else if constexpr (std::is_same_v)
+ _read_binary_data_(stream, idx_vec, needs_flipping);
+ else {
+ String1K err;
+
+ err.printf(
+ "read_binary_(): ERROR: Type '%s' is not supported for index",
+ col_type);
+ throw DataFrameError(err.c_str());
+ }
+ load_index(std::move(idx_vec));
+
+ for (uint16_t i = 0; i < col_num; ++i) {
+ stream.read(col_name, sizeof(col_name));
+ stream.read(col_type, sizeof(col_type));
+
+ if (! std::strcmp(col_type, "string")) {
+ ColumnVecType vec;
+
+ _read_binary_string_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if (! std::strcmp(col_type, "DateTime")) {
+ ColumnVecType vec;
+
+ _read_binary_datetime_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if (! std::strcmp(col_type, "float")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "double")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "short")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "ushort")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "int")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "uint")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "long")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "ulong")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "longlong")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "ulonglong")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "char")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "uchar")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else if ( ! std::strcmp(col_type, "bool")) {
+ ColumnVecType vec;
+
+ _read_binary_data_(stream, vec, needs_flipping);
+ load_column(col_name, std::move(vec),
+ nan_policy::dont_pad_with_nans);
+ }
+ else {
+ String1K err;
+
+ err.printf("read_binary_(): ERROR: Type '%s' is not supported",
+ col_type);
+ throw DataFrameError(err.c_str());
+ }
+ }
+}
+
+// ----------------------------------------------------------------------------
+
template
bool DataFrame::
read (const char *file_name,
@@ -1279,7 +1476,7 @@ read (const char *file_name,
size_type num_rows) {
std::ifstream stream;
- const IOStreamOpti io_opti(stream, file_name);
+ const IOStreamOpti io_opti(stream, file_name, iof == io_format::binary);
if (stream.fail()) [[unlikely]] {
String1K err;
@@ -1306,7 +1503,7 @@ read (S &in_s,
static_assert(std::is_base_of, DataVec>::value,
"Only a StdDataFrame can call read()");
- if (iof == io_format::csv) [[likely]] {
+ if (iof == io_format::csv) {
if (starting_row != 0 ||
num_rows != std::numeric_limits::max()) [[unlikely]]
throw NotImplemented("read(): Reading files in chunks is currently"
@@ -1325,6 +1522,15 @@ read (S &in_s,
read_json_ (in_s, columns_only);
}
+ else if (iof == io_format::binary) {
+ if (columns_only || starting_row != 0 ||
+ num_rows != std::numeric_limits::max()) [[unlikely]]
+ throw NotImplemented("read(): Reading columns only or in chunks "
+ "currently not implemented for "
+ "io_format::binary");
+
+ read_binary_ (in_s);
+ }
else
throw NotImplemented("read(): This io_format is not implemented");
diff --git a/include/DataFrame/Internals/DataFrame_standalone.tcc b/include/DataFrame/Internals/DataFrame_standalone.tcc
index cad56738..97090aa3 100644
--- a/include/DataFrame/Internals/DataFrame_standalone.tcc
+++ b/include/DataFrame/Internals/DataFrame_standalone.tcc
@@ -30,9 +30,11 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#pragma once
#include
+#include
#include
#include
+#include
#include
#include
#include
@@ -881,6 +883,210 @@ inline static S &_write_csv_df_index_(S &o, unsigned char value) {
// ----------------------------------------------------------------------------
+template
+inline static STRM &_write_binary_string_(STRM &strm, const V &str_vec) {
+
+ char buffer[32];
+
+ std::strncpy(buffer, "string", sizeof(buffer));
+ strm.write(buffer, sizeof(buffer));
+
+ const uint64_t vec_size = str_vec.size();
+
+ strm.write(reinterpret_cast(&vec_size), sizeof(vec_size));
+
+ // It is better for compression, if you write the alike data together
+ //
+ for (const auto &str : str_vec) {
+ const uint16_t str_sz = static_cast(str.size());
+
+ strm.write(reinterpret_cast(&str_sz), sizeof(str_sz));
+ }
+ for (const auto &str : str_vec)
+ strm.write(str.data(), str.size() * sizeof(char));
+
+ return (strm);
+}
+
+// ----------------------------------------------------------------------------
+
+template
+inline static STRM &_write_binary_data_(STRM &strm, const V &vec) {
+
+ using VecType = typename std::remove_reference::type;
+ using ValueType = typename VecType::value_type;
+
+ char buffer[32];
+ const auto &citer = _typeinfo_name_.find(typeid(ValueType));
+
+ if (citer != _typeinfo_name_.end()) [[likely]]
+ std::strncpy(buffer, citer->second, sizeof(buffer));
+ else
+ std::strncpy(buffer, "N/A", sizeof(buffer));
+ strm.write(buffer, sizeof(buffer));
+
+ const uint64_t vec_size = vec.size();
+
+ strm.write(reinterpret_cast(&vec_size), sizeof(vec_size));
+ if constexpr (std::is_same_v) {
+ for (const auto &b : vec) {
+ const bool bval = b;
+
+ strm.write(reinterpret_cast(&bval), sizeof(bool));
+ }
+ }
+ else {
+ // Views don't have the data() method
+ //
+ constexpr bool has_data_method =
+ requires(const VecType &v) { v.data(); };
+
+ if constexpr (has_data_method) {
+ strm.write(reinterpret_cast(vec.data()),
+ vec_size * sizeof(ValueType));
+ }
+ else {
+ for (std::size_t i = 0; i < vec_size; ++i)
+ strm.write(reinterpret_cast(&(vec[i])),
+ sizeof(ValueType));
+ }
+ }
+
+ return (strm);
+}
+
+// ----------------------------------------------------------------------------
+
+template
+inline static STRM &_write_binary_datetime_(STRM &strm, const V &dt_vec) {
+
+ char buffer[32];
+
+ std::strncpy(buffer, "DateTime", sizeof(buffer));
+ strm.write(buffer, sizeof(buffer));
+
+ const uint64_t vec_size = dt_vec.size();
+
+ strm.write(reinterpret_cast(&vec_size), sizeof(vec_size));
+ for (const auto &dt : dt_vec) {
+ const double val = static_cast(dt);
+
+ strm.write(reinterpret_cast(&val), sizeof(val));
+ }
+
+ return (strm);
+}
+
+// ----------------------------------------------------------------------------
+
+template
+inline static STRM &
+_read_binary_string_(STRM &strm, V &str_vec, bool needs_flipping) {
+
+ uint64_t vec_size { 0 };
+
+ strm.read(reinterpret_cast(&vec_size), sizeof(vec_size));
+ if (needs_flipping)
+ vec_size =
+ SwapBytes { }(vec_size);
+
+ std::vector sizes (vec_size, 0);
+
+ strm.read(reinterpret_cast(sizes.data()),
+ vec_size * sizeof(uint16_t));
+ if (needs_flipping) {
+ SwapBytes swaper { };
+
+ for (auto &s : sizes)
+ s = swaper(s);
+ }
+
+ // Now read the strings
+ //
+ str_vec.reserve(vec_size);
+ for (const auto s : sizes) {
+ std::string str (std::size_t(s), 0);
+
+ strm.read(str.data(), s * sizeof(char));
+ str_vec.emplace_back(std::move(str));
+ }
+
+ return (strm);
+}
+
+// ----------------------------------------------------------------------------
+
+template
+inline static STRM &
+_read_binary_data_(STRM &strm, V &vec, bool needs_flipping) {
+
+ using VecType = typename std::remove_reference::type;
+ using ValueType = typename VecType::value_type;
+
+ uint64_t vec_size { 0 };
+
+ strm.read(reinterpret_cast(&vec_size), sizeof(vec_size));
+ if (needs_flipping)
+ vec_size =
+ SwapBytes { }(vec_size);
+
+ if constexpr (std::is_same_v) {
+ vec.reserve(vec_size);
+ for (uint64_t i = 0; i < vec_size; ++i) {
+ bool val;
+
+ strm.read(reinterpret_cast(&val), sizeof(val));
+ vec.push_back(val);
+ }
+ }
+ else {
+ vec.resize(vec_size);
+ strm.read(reinterpret_cast(vec.data()),
+ vec_size * sizeof(ValueType));
+ if (needs_flipping) flip_endianness(vec);
+ }
+
+ return (strm);
+}
+
+// ----------------------------------------------------------------------------
+
+template
+inline static STRM &
+_read_binary_datetime_(STRM &strm, V &dt_vec, bool needs_flipping) {
+
+ uint64_t vec_size { 0 };
+
+ strm.read(reinterpret_cast(&vec_size), sizeof(vec_size));
+ if (needs_flipping)
+ vec_size =
+ SwapBytes { }(vec_size);
+
+ SwapBytes swaper { };
+
+ dt_vec.reserve(vec_size);
+ for (uint64_t i = 0; i < vec_size; ++i) {
+ double val { 0 };
+
+ strm.read(reinterpret_cast(&val), sizeof(val));
+ if (needs_flipping) val = swaper(val);
+
+ DateTime dt;
+ const DateTime::EpochType tm =
+ static_cast(val);
+ const DateTime::NanosecondType nano =
+ static_cast(
+ (val - static_cast(tm)) * 1'000'000'000.0);
+
+ dt.set_time(tm, nano);
+ dt_vec.emplace_back(dt);
+ }
+
+ return (strm);
+}
+
+// ----------------------------------------------------------------------------
+
//
// Specializing std::hash for tuples
//
@@ -1203,7 +1409,7 @@ struct _LikeClauseUtil_ {
//
// NOTE: This could be, in some cases, n-squared. But it is pretty fast with
// moderately sized strings. I have not tested this with huge/massive
-// strings.
+// strings.
//
static inline bool
_like_clause_compare_(const char *pattern,
diff --git a/include/DataFrame/Internals/DataFrame_write.tcc b/include/DataFrame/Internals/DataFrame_write.tcc
index 8fed069b..da429500 100644
--- a/include/DataFrame/Internals/DataFrame_write.tcc
+++ b/include/DataFrame/Internals/DataFrame_write.tcc
@@ -28,6 +28,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include
+#include
#include
// ----------------------------------------------------------------------------
@@ -45,7 +46,7 @@ write(const char *file_name,
long max_recs) const {
std::ofstream stream;
- const IOStreamOpti io_opti(stream, file_name);
+ const IOStreamOpti io_opti(stream, file_name, iof == io_format::binary);
if (stream.fail()) [[unlikely]] {
String1K err;
@@ -53,7 +54,8 @@ write(const char *file_name,
err.printf("write(): ERROR: Unable to open file '%s'", file_name);
throw DataFrameError(err.c_str());
}
- write(stream, iof, precision, columns_only, max_recs);
+ write
+ (stream, iof, precision, columns_only, max_recs);
return (true);
}
@@ -83,7 +85,8 @@ write(S &o,
if (iof != io_format::csv &&
iof != io_format::json &&
- iof != io_format::csv2)
+ iof != io_format::csv2 &&
+ iof != io_format::binary)
throw NotImplemented("write(): This io_format is not implemented");
bool need_pre_comma = false;
@@ -193,10 +196,38 @@ write(S &o,
o << '\n';
}
}
+ else if (iof == io_format::binary) {
+ const auto ed = get_system_endian();
+
+ o.write(reinterpret_cast(&ed), sizeof(ed));
+
+ const uint16_t col_num = static_cast(column_list_.size());
+
+ o.write(reinterpret_cast(&col_num), sizeof(col_num));
+
+ print_binary_functor_ idx_functor (DF_INDEX_COL_NAME,
+ o,
+ start_row,
+ end_row);
+
+ idx_functor(indices_);
+
+ const SpinGuard guard(lock_);
+
+ for (const auto &[name, idx] : column_list_) [[likely]] {
+ print_binary_functor_ functor (name.c_str(),
+ o,
+ start_row,
+ end_row);
+
+ data_[idx].change(functor);
+ }
+ }
if (iof == io_format::json)
o << "\n}";
- o << std::endl;
+ // if (iof != io_format::binary)
+ // o << std::endl;
return (true);
}
diff --git a/include/DataFrame/Utils/Endianness.h b/include/DataFrame/Utils/Endianness.h
new file mode 100644
index 00000000..a1a0f0a4
--- /dev/null
+++ b/include/DataFrame/Utils/Endianness.h
@@ -0,0 +1,209 @@
+// Hossein Moein
+// May 14, 2024
+/*
+Copyright (c) 2019-2026, Hossein Moein
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+* Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+* Neither the name of Hossein Moein and/or the DataFrame nor the
+ names of its contributors may be used to endorse or promote products
+ derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL Hossein Moein BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+*/
+
+#pragma once
+
+#include
+
+#include
+#include
+#include
+
+// ----------------------------------------------------------------------------
+
+namespace hmdf
+{
+
+enum class endians : unsigned char {
+
+ little_endian = 1,
+ big_endian = 2,
+ mixed_endian = 3,
+};
+
+// ----------------------------------------------------------------------------
+
+template
+struct SwapBytes {
+
+ inline T operator()(T) const {
+
+ throw NotImplemented("SwapBytes: Type/Size");
+ }
+};
+
+// Specialisations
+//
+template
+struct SwapBytes {
+
+ inline T operator()(T value) const { return (value); }
+};
+
+template
+struct SwapBytes {
+
+ inline T operator()(T value) const {
+
+ return (((value >> 8) & 0xff) |
+ ((value & 0xff) << 8));
+ }
+};
+
+template
+struct SwapBytes {
+
+ inline T operator()(T value) const {
+
+ return (((value & 0xff000000) >> 24) |
+ ((value & 0x00ff0000) >> 8) |
+ ((value & 0x0000ff00) << 8) |
+ ((value & 0x000000ff) << 24));
+ }
+};
+
+template<>
+struct SwapBytes {
+
+ inline float operator()(float value) const {
+
+ const uint32_t layout =
+ SwapBytes{ }(*((uint32_t *) &value));
+
+ return (*((float *) &layout));
+
+ }
+};
+
+template
+struct SwapBytes {
+
+ inline T operator()(T value) const {
+
+ return (((value & 0xff00000000000000ULL) >> 56) |
+ ((value & 0x00ff000000000000ULL) >> 40) |
+ ((value & 0x0000ff0000000000ULL) >> 24) |
+ ((value & 0x000000ff00000000ULL) >> 8) |
+ ((value & 0x00000000ff000000ULL) << 8) |
+ ((value & 0x0000000000ff0000ULL) << 24) |
+ ((value & 0x000000000000ff00ULL) << 40) |
+ ((value & 0x00000000000000ffULL) << 56));
+ }
+};
+
+template<>
+struct SwapBytes {
+
+ inline double operator()(double value) const {
+
+ const uint64_t layout =
+ SwapBytes{ }(*((uint64_t *) &value));
+
+ return (*((double *) &layout));
+
+ }
+};
+
+/*
+template
+struct SwapBytes {
+
+ inline T operator()(T value) const {
+
+ return (((value & 0xff000000000000000000000000000000ULL) >> 120) |
+ ((value & 0x00ff0000000000000000000000000000ULL) >> 104) |
+ ((value & 0x0000ff00000000000000000000000000ULL) >> 88) |
+ ((value & 0x000000ff000000000000000000000000ULL) >> 72) |
+ ((value & 0x00000000ff0000000000000000000000ULL) >> 56) |
+ ((value & 0x0000000000ff00000000000000000000ULL) >> 40) |
+ ((value & 0x000000000000ff000000000000000000ULL) >> 24) |
+ ((value & 0x00000000000000ff0000000000000000ULL) >> 8) |
+ ((value & 0x0000000000000000ff00000000000000ULL) << 8) |
+ ((value & 0x000000000000000000ff000000000000ULL) << 24) |
+ ((value & 0x00000000000000000000ff0000000000ULL) << 40) |
+ ((value & 0x0000000000000000000000ff00000000ULL) << 56) |
+ ((value & 0x000000000000000000000000ff000000ULL) << 72) |
+ ((value & 0x00000000000000000000000000ff0000ULL) << 88) |
+ ((value & 0x0000000000000000000000000000ff00ULL) << 104) |
+ ((value & 0x000000000000000000000000000000ffULL) << 120));
+ }
+};
+
+template<>
+struct SwapBytes {
+
+ inline long double operator()(long double value) const {
+
+ const uint128_t layout =
+ SwapBytes{ }(*((uint128_t *) &value));
+
+ return (*((long double *) &layout));
+
+ }
+};
+*/
+
+// ----------------------------------------------------------------------------
+
+static inline
+constexpr endians get_system_endian() {
+
+ if constexpr (std::endian::native == std::endian::big)
+ return (endians::big_endian);
+ else if constexpr (std::endian::native == std::endian::little)
+ return (endians::little_endian);
+ else
+ return (endians::mixed_endian);
+}
+
+// ----------------------------------------------------------------------------
+
+template
+static inline
+void flip_endianness(V &vec) {
+
+ using VecType = typename std::remove_reference::type;
+ using ValueType = typename VecType::value_type;
+
+ if constexpr (sizeof(ValueType) < 2) return;
+
+ const SwapBytes swaper { };
+
+ for (auto &val : vec) val = swaper(val);
+}
+
+} // namespace hmdf
+
+// ----------------------------------------------------------------------------
+
+// Local Variables:
+// mode:C++
+// tab-width:4
+// c-basic-offset:4
+// End:
diff --git a/include/DataFrame/Utils/FixedSizeString.h b/include/DataFrame/Utils/FixedSizeString.h
index b9056cf3..0ea6f4c3 100644
--- a/include/DataFrame/Utils/FixedSizeString.h
+++ b/include/DataFrame/Utils/FixedSizeString.h
@@ -291,6 +291,12 @@ class VirtualString {
//
[[nodiscard]] inline const_pointer
c_str () const noexcept { return (string_); }
+
+ [[nodiscard]] inline const_pointer
+ data () const noexcept { return (string_); }
+ [[nodiscard]] inline pointer
+ data () noexcept { return (string_); }
+
[[nodiscard]] inline const_pointer
sub_c_str (size_type offset) const noexcept {
diff --git a/include/DataFrame/Utils/Utils.h b/include/DataFrame/Utils/Utils.h
index 60877c46..19bb977f 100644
--- a/include/DataFrame/Utils/Utils.h
+++ b/include/DataFrame/Utils/Utils.h
@@ -200,7 +200,7 @@ get_nan() { return (std::numeric_limits::quiet_NaN()); }
template<>
inline DateTime
-get_nan() { return (DateTime { 19700101 }); }
+get_nan() { return (DateTime { DateTime::DateType(19700101) }); }
// ----------------------------------------------------------------------------
@@ -347,17 +347,21 @@ shift_left(V &vec, std::size_t n) {
// ----------------------------------------------------------------------------
-template
+template
struct IOStreamOpti {
- IOStreamOpti (STR &stream, const char *file_name)
+ IOStreamOpti (STR &stream, const char *file_name, bool binary = false)
: stream_(stream),
tie_(std::cin.tie(nullptr)),
sync_(std::ios_base::sync_with_stdio(false)) {
stream_.rdbuf()->pubsetbuf(buffer_, SIZ);
- if (file_name && ! stream_.is_open())
- stream_.open(file_name);
+ if (file_name && ! stream_.is_open()) {
+ if (! binary)
+ stream_.open(file_name);
+ else
+ stream_.open(file_name, std::ios::binary);
+ }
}
~IOStreamOpti () {
diff --git a/src/CommonMakefile.mk b/src/CommonMakefile.mk
index dca3883d..77bd60ba 100644
--- a/src/CommonMakefile.mk
+++ b/src/CommonMakefile.mk
@@ -73,7 +73,8 @@ HEADERS = $(LOCAL_INCLUDE_DIR)/DataFrame/Vectors/HeteroVector.h \
$(LOCAL_INCLUDE_DIR)/DataFrame/Utils/Concepts.h \
$(LOCAL_INCLUDE_DIR)/DataFrame/Utils/FixedSizeString.h \
$(LOCAL_INCLUDE_DIR)/DataFrame/Utils/FixedSizePriorityQueue.h \
- $(LOCAL_INCLUDE_DIR)/DataFrame/Utils/AlignedAllocator.h
+ $(LOCAL_INCLUDE_DIR)/DataFrame/Utils/AlignedAllocator.h \
+ $(LOCAL_INCLUDE_DIR)/DataFrame/Utils/Endianness.h
LIB_NAME = DataFrame
TARGET_LIB = $(LOCAL_LIB_DIR)/lib$(LIB_NAME).a
diff --git a/test/dataframe_tester.cc b/test/dataframe_tester.cc
index 094b298d..8cac9408 100644
--- a/test/dataframe_tester.cc
+++ b/test/dataframe_tester.cc
@@ -1256,11 +1256,11 @@ static void test_dataframe_with_datetime() {
std::cout << "\nTesting DataFrame with DateTime ..." << std::endl;
- DateTime dt(20010102);
- StlVecType idx;
- StlVecType d1;
- StlVecType i1;
- StlVecType s1;
+ DateTime dt (DateTime::DateType(20010102));
+ StlVecType idx;
+ StlVecType d1;
+ StlVecType i1;
+ StlVecType s1;
idx.reserve(20);
d1.reserve(20);
diff --git a/test/dataframe_tester_2.cc b/test/dataframe_tester_2.cc
index 21f98f8b..6389bd95 100644
--- a/test/dataframe_tester_2.cc
+++ b/test/dataframe_tester_2.cc
@@ -2623,10 +2623,10 @@ static void test_DT_IBM_data() {
assert(df.get_column("IBM_Open")[0] == 98.4375);
assert(df.get_column("IBM_Close")[18] == 97.875);
- assert(df.get_index()[18] == DateTime(20001128));
+ assert(df.get_index()[18] == DateTime(DateTime::DateType(20001128)));
assert(fabs(df.get_column("IBM_High")[5030] - 111.8) < 0.001);
assert(df.get_column("IBM_Volume")[5022] == 21501100L);
- assert(df.get_index()[5020] == DateTime(20201016));
+ assert(df.get_index()[5020] == DateTime(DateTime::DateType(20201016)));
}
catch (const DataFrameError &ex) {
std::cout << ex.what() << std::endl;
diff --git a/test/dataframe_tester_3.cc b/test/dataframe_tester_3.cc
index a951f5b7..f793228a 100644
--- a/test/dataframe_tester_3.cc
+++ b/test/dataframe_tester_3.cc
@@ -32,6 +32,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#include
#include
+#include
#include
#include
@@ -3771,6 +3772,249 @@ static void test_EhlersBandPassFilterVisitor() {
// ----------------------------------------------------------------------------
+static void test_writing_binary() {
+
+ std::cout << "\nTesting test_writing_binary{ } ..." << std::endl;
+
+ typedef StdDataFrame64 StrDataFrame;
+
+ StrDataFrame ibm;
+ StrDataFrame ibm_csv;
+ StrDataFrame ibm_csv2;
+ StrDataFrame ibm_dat;
+ StrDataFrame ibm_vw_dat;
+ StrDataFrame ibm_vw_json;
+
+ try {
+ ibm.read("SHORT_IBM.csv", io_format::csv2);
+
+ ibm.write("./SHORT_IBM_dup.csv", io_format::csv);
+ ibm.write("./SHORT_IBM_dup.csv2", io_format::csv2);
+ ibm.write("./SHORT_IBM_dup.dat", io_format::binary);
+
+ auto vw =
+ ibm.get_view(
+ { "IBM_Open", "IBM_High", "IBM_Close", "IBM_Volume" });
+
+ vw.write("./FROM_VW_SHORT_IBM.csv", io_format::csv);
+ vw.write("./FROM_VW_SHORT_IBM.csv2", io_format::csv2);
+ vw.write("./FROM_VW_SHORT_IBM.dat", io_format::binary);
+ vw.write("./FROM_VW_SHORT_IBM.json", io_format::json);
+
+ ibm_csv.read("./SHORT_IBM_dup.csv", io_format::csv);
+ ibm_csv2.read("./SHORT_IBM_dup.csv2", io_format::csv2);
+ ibm_dat.read("./SHORT_IBM_dup.dat", io_format::binary);
+ ibm_vw_dat.read("./FROM_VW_SHORT_IBM.dat", io_format::binary);
+ ibm_vw_json.read("./FROM_VW_SHORT_IBM.json", io_format::json);
+
+ assert((ibm.get_index().size() == ibm_dat.get_index().size()));
+ assert((ibm.get_index().size() == ibm_csv.get_index().size()));
+ assert((ibm.get_index().size() == ibm_csv2.get_index().size()));
+ assert((ibm.get_index().size() == ibm_vw_dat.get_index().size()));
+ assert((ibm.get_index().size() == ibm_vw_json.get_index().size()));
+
+ assert((ibm.get_index().front() == ibm_dat.get_index().front()));
+ assert((ibm.get_index().front() == ibm_csv.get_index().front()));
+ assert((ibm.get_index().front() == ibm_csv2.get_index().front()));
+ assert((ibm.get_index().front() == ibm_vw_dat.get_index().front()));
+ assert((ibm.get_index().front() == ibm_vw_json.get_index().front()));
+
+ assert((ibm.get_index().back() == ibm_dat.get_index().back()));
+ assert((ibm.get_index().back() == ibm_csv.get_index().back()));
+ assert((ibm.get_index().back() == ibm_csv2.get_index().back()));
+ assert((ibm.get_index().back() == ibm_vw_dat.get_index().back()));
+ assert((ibm.get_index().back() == ibm_vw_json.get_index().back()));
+
+ assert((ibm.get_index()[1000] == ibm_dat.get_index()[1000]));
+ assert((ibm.get_index()[1000] == ibm_csv.get_index()[1000]));
+ assert((ibm.get_index()[1000] == ibm_csv2.get_index()[1000]));
+ assert((ibm.get_index()[1000] == ibm_vw_dat.get_index()[1000]));
+ assert((ibm.get_index()[1000] == ibm_vw_json.get_index()[1000]));
+
+ assert((ibm.get_column("IBM_Open").front() ==
+ ibm_dat.get_column("IBM_Open").front()));
+ assert((ibm.get_column("IBM_Open").front() ==
+ ibm_csv.get_column("IBM_Open").front()));
+ assert((ibm.get_column("IBM_Open").front() ==
+ ibm_csv2.get_column("IBM_Open").front()));
+ assert((ibm.get_column("IBM_Open").front() ==
+ ibm_vw_dat.get_column("IBM_Open").front()));
+ assert((ibm.get_column("IBM_Open").front() ==
+ ibm_vw_json.get_column("IBM_Open").front()));
+
+ assert((ibm.get_column("IBM_Open").back() ==
+ ibm_dat.get_column("IBM_Open").back()));
+ assert((ibm.get_column("IBM_Open").back() ==
+ ibm_csv.get_column("IBM_Open").back()));
+ assert((ibm.get_column("IBM_Open").back() ==
+ ibm_csv2.get_column("IBM_Open").back()));
+ assert((ibm.get_column("IBM_Open").back() ==
+ ibm_vw_dat.get_column("IBM_Open").back()));
+ assert((ibm.get_column("IBM_Open").back() ==
+ ibm_vw_json.get_column("IBM_Open").back()));
+
+ assert((ibm.get_column("IBM_Open")[850] ==
+ ibm_dat.get_column("IBM_Open")[850]));
+ assert((ibm.get_column("IBM_Open")[850] ==
+ ibm_csv.get_column("IBM_Open")[850]));
+ assert((ibm.get_column("IBM_Open")[850] ==
+ ibm_csv2.get_column("IBM_Open")[850]));
+ assert((ibm.get_column("IBM_Open")[850] ==
+ ibm_vw_dat.get_column("IBM_Open")[850]));
+ assert((ibm.get_column("IBM_Open")[850] ==
+ ibm_vw_json.get_column("IBM_Open")[850]));
+
+ assert((ibm.get_column("IBM_High").front() ==
+ ibm_dat.get_column("IBM_High").front()));
+ assert((ibm.get_column("IBM_High").front() ==
+ ibm_csv.get_column("IBM_High").front()));
+ assert((ibm.get_column("IBM_High").front() ==
+ ibm_csv2.get_column("IBM_High").front()));
+ assert((ibm.get_column("IBM_High").front() ==
+ ibm_vw_dat.get_column("IBM_High").front()));
+ assert((ibm.get_column("IBM_High").front() ==
+ ibm_vw_json.get_column("IBM_High").front()));
+
+ assert((ibm.get_column("IBM_High").back() ==
+ ibm_dat.get_column("IBM_High").back()));
+ assert((ibm.get_column("IBM_High").back() ==
+ ibm_csv.get_column("IBM_High").back()));
+ assert((ibm.get_column("IBM_High").back() ==
+ ibm_csv2.get_column("IBM_High").back()));
+ assert((ibm.get_column("IBM_High").back() ==
+ ibm_vw_dat.get_column("IBM_High").back()));
+ assert((ibm.get_column("IBM_High").back() ==
+ ibm_vw_json.get_column("IBM_High").back()));
+
+ assert((ibm.get_column("IBM_High")[850] ==
+ ibm_dat.get_column("IBM_High")[850]));
+ assert((ibm.get_column("IBM_High")[850] ==
+ ibm_csv.get_column("IBM_High")[850]));
+ assert((ibm.get_column("IBM_High")[850] ==
+ ibm_csv2.get_column("IBM_High")[850]));
+ assert((ibm.get_column("IBM_High")[850] ==
+ ibm_vw_dat.get_column |