Skip to content

Commit

Permalink
Reduce memory usage during conda build (#1649)
Browse files Browse the repository at this point in the history
Add more translation units for binary operators to reduce memory usage.

The build on conda was failing in arcticdb-feedstock as it was using a
lot of memory. The feedstock runners only have `6.7GB` of RAM. Splitting
the binary operators in multiple translation units reduces memory usage
and fixes the issue.

The issue was also observed in ArcticDB CI.

#### Checklist

<details>
  <summary>
   Checklist for code changes...
  </summary>
 
- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->
  • Loading branch information
muhammadhamzasajjad authored Jun 26, 2024
1 parent c81f8ad commit 9b2cbb9
Show file tree
Hide file tree
Showing 11 changed files with 94 additions and 11 deletions.
8 changes: 7 additions & 1 deletion cpp/arcticdb/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -439,9 +439,15 @@ set(arcticdb_srcs
processing/operation_dispatch_unary.cpp
processing/operation_dispatch_binary.cpp
processing/operation_dispatch_binary_eq.cpp
processing/operation_dispatch_binary_neq.cpp
processing/operation_dispatch_binary_gt.cpp
processing/operation_dispatch_binary_gte.cpp
processing/operation_dispatch_binary_lt.cpp
processing/operation_dispatch_binary_operator.cpp
processing/operation_dispatch_binary_lte.cpp
processing/operation_dispatch_binary_operator_plus.cpp
processing/operation_dispatch_binary_operator_minus.cpp
processing/operation_dispatch_binary_operator_times.cpp
processing/operation_dispatch_binary_operator_divide.cpp
processing/sorted_aggregation.cpp
processing/unsorted_aggregation.cpp
python/python_to_tensor_frame.cpp
Expand Down
1 change: 0 additions & 1 deletion cpp/arcticdb/processing/operation_dispatch_binary_eq.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,5 @@
namespace arcticdb {

template VariantData visit_binary_comparator<EqualsOperator>(const VariantData&, const VariantData&, EqualsOperator&&);
template VariantData visit_binary_comparator<NotEqualsOperator>(const VariantData&, const VariantData&, NotEqualsOperator&&);

}
1 change: 0 additions & 1 deletion cpp/arcticdb/processing/operation_dispatch_binary_gt.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,5 @@
namespace arcticdb {

template VariantData visit_binary_comparator<GreaterThanOperator>(const VariantData&, const VariantData&, GreaterThanOperator&&);
template VariantData visit_binary_comparator<GreaterThanEqualsOperator>(const VariantData&, const VariantData&, GreaterThanEqualsOperator&&);

}
15 changes: 15 additions & 0 deletions cpp/arcticdb/processing/operation_dispatch_binary_gte.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/*
* Copyright 2023 Man Group Operations Limited
*
* Use of this software is governed by the Business Source License 1.1 included in the file licenses/BSL.txt.
*
* As of the Change Date specified in that file, in accordance with the Business Source License, use of this software will be governed by the Apache License, version 2.0.
*/

#include <arcticdb/processing/operation_dispatch_binary.hpp>

namespace arcticdb {

template VariantData visit_binary_comparator<GreaterThanEqualsOperator>(const VariantData&, const VariantData&, GreaterThanEqualsOperator&&);

}
1 change: 0 additions & 1 deletion cpp/arcticdb/processing/operation_dispatch_binary_lt.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,5 @@
namespace arcticdb {

template VariantData visit_binary_comparator<LessThanOperator>(const VariantData&, const VariantData&, LessThanOperator&&);
template VariantData visit_binary_comparator<LessThanEqualsOperator>(const VariantData&, const VariantData&, LessThanEqualsOperator&&);

}
15 changes: 15 additions & 0 deletions cpp/arcticdb/processing/operation_dispatch_binary_lte.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/*
* Copyright 2023 Man Group Operations Limited
*
* Use of this software is governed by the Business Source License 1.1 included in the file licenses/BSL.txt.
*
* As of the Change Date specified in that file, in accordance with the Business Source License, use of this software will be governed by the Apache License, version 2.0.
*/

#include <arcticdb/processing/operation_dispatch_binary.hpp>

namespace arcticdb {

template VariantData visit_binary_comparator<LessThanEqualsOperator>(const VariantData&, const VariantData&, LessThanEqualsOperator&&);

}
15 changes: 15 additions & 0 deletions cpp/arcticdb/processing/operation_dispatch_binary_neq.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/*
* Copyright 2023 Man Group Operations Limited
*
* Use of this software is governed by the Business Source License 1.1 included in the file licenses/BSL.txt.
*
* As of the Change Date specified in that file, in accordance with the Business Source License, use of this software will be governed by the Apache License, version 2.0.
*/

#include <arcticdb/processing/operation_dispatch_binary.hpp>

namespace arcticdb {

template VariantData visit_binary_comparator<NotEqualsOperator>(const VariantData&, const VariantData&, NotEqualsOperator&&);

}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright 2023 Man Group Operations Limited
* Copyright 2024 Man Group Operations Limited
*
* Use of this software is governed by the Business Source License 1.1 included in the file licenses/BSL.txt.
*
Expand All @@ -9,10 +9,5 @@
#include <arcticdb/processing/operation_dispatch_binary.hpp>

namespace arcticdb {

template VariantData visit_binary_operator<PlusOperator>(const VariantData&, const VariantData&, PlusOperator&&);
template VariantData visit_binary_operator<MinusOperator>(const VariantData&, const VariantData&, MinusOperator&&);
template VariantData visit_binary_operator<TimesOperator>(const VariantData&, const VariantData&, TimesOperator&&);
template VariantData visit_binary_operator<DivideOperator>(const VariantData&, const VariantData&, DivideOperator&&);

}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/*
* Copyright 2024 Man Group Operations Limited
*
* Use of this software is governed by the Business Source License 1.1 included in the file licenses/BSL.txt.
*
* As of the Change Date specified in that file, in accordance with the Business Source License, use of this software will be governed by the Apache License, version 2.0.
*/

#include <arcticdb/processing/operation_dispatch_binary.hpp>

namespace arcticdb {
template VariantData visit_binary_operator<MinusOperator>(const VariantData&, const VariantData&, MinusOperator&&);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
/*
* Copyright 2024 Man Group Operations Limited
*
* Use of this software is governed by the Business Source License 1.1 included in the file licenses/BSL.txt.
*
* As of the Change Date specified in that file, in accordance with the Business Source License, use of this software will be governed by the Apache License, version 2.0.
*/

#include <arcticdb/processing/operation_dispatch_binary.hpp>

namespace arcticdb {

template VariantData visit_binary_operator<PlusOperator>(const VariantData&, const VariantData&, PlusOperator&&);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/*
* Copyright 2024 Man Group Operations Limited
*
* Use of this software is governed by the Business Source License 1.1 included in the file licenses/BSL.txt.
*
* As of the Change Date specified in that file, in accordance with the Business Source License, use of this software will be governed by the Apache License, version 2.0.
*/

#include <arcticdb/processing/operation_dispatch_binary.hpp>

namespace arcticdb {
template VariantData visit_binary_operator<TimesOperator>(const VariantData&, const VariantData&, TimesOperator&&);
}

0 comments on commit 9b2cbb9

Please sign in to comment.