Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use memory management functions #22

Draft
wants to merge 40 commits into
base: master
Choose a base branch
from
Draft

Use memory management functions #22

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
e2d1fff
fix import error
Fadila82 Jul 4, 2022
cbf3877
free errors
Fadila82 Jul 6, 2022
fac90ec
access: deallocate unused c struct
Fadila82 Jul 6, 2022
09d173d
free unused structs
Fadila82 Jul 7, 2022
c0e0c9c
fix codacy issues
Fadila82 Jul 7, 2022
a8da7bb
linting
Fadila82 Jul 8, 2022
7f74da2
remove duplicate code
Fadila82 Aug 8, 2022
80ccdc5
correct according to codacy checks
Fadila82 Aug 8, 2022
a19bcaa
update satellite address
Fadila82 Sep 6, 2022
dbbed15
argstypes replaced by argtypes
Fadila82 Oct 29, 2022
2fd62c8
do not free upload result
Fadila82 Oct 30, 2022
b015729
test
Fadila82 Oct 31, 2022
e3e6934
test
Fadila82 Oct 31, 2022
0711d1e
test
Fadila82 Oct 31, 2022
6547aa2
use unwrap_libuplink_result
Fadila82 Nov 3, 2022
ea0d41b
code clean-up
Fadila82 Nov 3, 2022
02721a7
code clean-up
Fadila82 Nov 3, 2022
493d6d6
code clean-up
Fadila82 Nov 3, 2022
d3c4f48
code clean-up
Fadila82 Nov 3, 2022
139c51a
code cleanup
Fadila82 Nov 4, 2022
2162e46
code cleanup
Fadila82 Nov 4, 2022
83607ab
code cleanup
Fadila82 Nov 4, 2022
12851fc
code cleanup
Fadila82 Nov 4, 2022
d083724
code cleanup
Fadila82 Nov 4, 2022
86e51ab
code cleanup
Fadila82 Nov 4, 2022
4f6c64f
add __del__ to free c memory
Fadila82 Nov 10, 2022
b8f98a2
add __del__ to free c memory
Fadila82 Nov 10, 2022
c30c650
typo
Fadila82 Nov 11, 2022
d4cbf8a
storj-sim for test
Fadila82 Nov 20, 2022
4021f41
storj-sim for test
Fadila82 Nov 20, 2022
30fc6ae
storj-sim for test
Fadila82 Nov 20, 2022
ed9b284
storj-sim for test
Fadila82 Nov 20, 2022
a9e605b
storj-sim for test
Fadila82 Nov 20, 2022
69035b7
storj-sim for test
Fadila82 Nov 20, 2022
9aba7aa
storj-sim for test
Fadila82 Nov 20, 2022
2a9579e
storj-sim for test
Fadila82 Nov 20, 2022
09e200d
storj-sim for test
Fadila82 Nov 20, 2022
888345e
ci back to original
Fadila82 Nov 22, 2022
a589059
code cleanup
Fadila82 Nov 24, 2022
90970c0
code cleanup
Fadila82 Nov 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion test/test_data/helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def __init__(self):
self.api_key = file_handle.read()
file_handle.close()

self.satellite = "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us-central-1.tardigrade.io:7777"
self.satellite = "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"
self.encryption_phrase = "test"

self.uplink = Uplink()
Expand Down
1 change: 0 additions & 1 deletion test/test_data/object_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
import string
import unittest

from uplink_python.errors import StorjException, ERROR_OBJECT_NOT_FOUND
from .helper import TestPy


Expand Down
77 changes: 61 additions & 16 deletions uplink_python/access.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ def derive_encryption_key(self, passphrase: str, salt: str):
ctypes.c_void_p,
ctypes.c_size_t]
self.uplink.m_libuplink.uplink_derive_encryption_key.restype = _EncryptionKeyResult
self.uplink.m_libuplink.uplink_free_encryption_key_result.argtypes = [_EncryptionKeyResult]
#
# prepare the input for the function
passphrase_ptr = ctypes.c_char_p(passphrase.encode('utf-8'))
Expand All @@ -80,9 +81,18 @@ def derive_encryption_key(self, passphrase: str, salt: str):
#
# if error occurred
if bool(encryption_key_result.error):
raise _storj_exception(encryption_key_result.error.contents.code,
encryption_key_result.error.contents.message.decode("utf-8"))
return encryption_key_result.encryption_key
error_code = encryption_key_result.error.contents.code
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is a pretty common pattern here. Also, though, it looks kind of fragile. Even if we are very careful this time, the next person to work on this code could easily mess something up.

It might be worth isolating this pattern to a few utility functions. Something like:

def unwrap_libuplink_result(result_object, finalizer, attribute_name):
    if bool(result_object.error):
        error_code = result_object.error.contents.code
        error_msg = result_object.error.contents.message.decode("utf-8")
        finalizer(result_object)
        raise _storj_exception(error_code, error_msg)
    result = getattr(result_object, attribute_name)
    finalizer(result_object)
    return result

def unwrap_encryption_key_result(result_object, uplink_handle):
    return unwrap_libuplink_result(result_object, uplink_handle.m_libuplink.uplink_free_encryption_key_result, 'encryption_key')

def unwrap_project_result(result_object, uplink_handle):
    return unwrap_libuplink_result(result_object, uplink_handle.m_libuplink.uplink_free_project_result, 'project')

# (and similar methods for StringResult, AccessResult, etc)

Then in this function you could replace this whole ending (everything from line 81 on) with:

    return unwrap_encryption_key_result(encryption_key_result, self.uplink)

The preexisting code in this repo is still pretty messy, though, so that could only help so much.

error_msg = encryption_key_result.error.contents.message.decode("utf-8")

self.uplink.m_libuplink.uplink_free_encryption_key_result(encryption_key_result)

raise _storj_exception(error_code, error_msg)

encryption_key = encryption_key_result.encryption_key

self.uplink.m_libuplink.uplink_free_encryption_key_result(encryption_key_result)

return encryption_key

def override_encryption_key(self, bucket_name: str, prefix: str, encryption_key):
"""
Expand Down Expand Up @@ -115,8 +125,7 @@ def override_encryption_key(self, bucket_name: str, prefix: str, encryption_key)
#
# if error occurred
if bool(error_result):
raise _storj_exception(error_result.contents.code,
error_result.contents.message.decode("utf-8"))
self.uplink.free_error_and_raise_exception(error_result)

def open_project(self):
"""
Expand All @@ -137,8 +146,14 @@ def open_project(self):
#
# if error occurred
if bool(project_result.error):
raise _storj_exception(project_result.error.contents.code,
project_result.error.contents.message.decode("utf-8"))
error_code = project_result.error.contents.code
error_msg = project_result.error.contents.message.decode("utf-8")

self.uplink.m_libuplink.uplink_free_project_result.argstypes = [_ProjectResult]
Fadila82 marked this conversation as resolved.
Show resolved Hide resolved
self.uplink.m_libuplink.uplink_free_project_result(project_result)

raise _storj_exception(error_code,error_msg)

return Project(project_result.project, self.uplink)

def config_open_project(self, config: Config):
Expand All @@ -159,6 +174,7 @@ def config_open_project(self, config: Config):
self.uplink.m_libuplink.uplink_config_open_project.argtypes =\
[_ConfigStruct, ctypes.POINTER(_AccessStruct)]
self.uplink.m_libuplink.uplink_config_open_project.restype = _ProjectResult
self.uplink.m_libuplink.uplink_free_project_result.argstypes = [_ProjectResult]
Fadila82 marked this conversation as resolved.
Show resolved Hide resolved
#
# prepare the input for the function
if config is None:
Expand All @@ -171,9 +187,19 @@ def config_open_project(self, config: Config):
#
# if error occurred
if bool(project_result.error):
raise _storj_exception(project_result.error.contents.code,
project_result.error.contents.message.decode("utf-8"))
return Project(project_result.project, self.uplink)
error_code = project_result.error.contents.code
error_msg = project_result.error.contents.message.decode("utf-8")

self.uplink.m_libuplink.uplink_free_project_result(project_result)

raise _storj_exception(error_code,error_msg)

project = Project(project_result.project, self.uplink)

self.uplink.m_libuplink.uplink_free_project_result(project_result)

return project


def serialize(self):
"""
Expand All @@ -189,15 +215,24 @@ def serialize(self):
# declare types of arguments and response of the corresponding golang function
self.uplink.m_libuplink.uplink_access_serialize.argtypes = [ctypes.POINTER(_AccessStruct)]
self.uplink.m_libuplink.uplink_access_serialize.restype = _StringResult
self.uplink.m_libuplink.uplink_free_string_result.argstypes = [_StringResult]
Fadila82 marked this conversation as resolved.
Show resolved Hide resolved
#
# get serialized access by calling the exported golang function
string_result = self.uplink.m_libuplink.uplink_access_serialize(self.access)
#
# if error occurred
if bool(string_result.error):
raise _storj_exception(string_result.error.contents.code,
string_result.error.contents.message.decode("utf-8"))
return string_result.string.decode("utf-8")
error_code = string_result.error.contents.code
error_msg = string_result.error.contents.message.decode("utf-8")

self.uplink.m_libuplink.uplink_free_string_result(string_result)
raise _storj_exception(error_code,error_msg)

to_return = string_result.string.decode("utf-8")

self.uplink.m_libuplink.uplink_free_string_result(string_result)

return to_return

def share(self, permission: Permission = None, shared_prefix: [SharePrefix] = None):
"""
Expand Down Expand Up @@ -227,6 +262,7 @@ def share(self, permission: Permission = None, shared_prefix: [SharePrefix] = No
ctypes.POINTER(_SharePrefixStruct),
ctypes.c_size_t]
self.uplink.m_libuplink.uplink_access_share.restype = _AccessResult
self.uplink.m_libuplink.uplink_free_access_result.argstypes = [_AccessResult]
Fadila82 marked this conversation as resolved.
Show resolved Hide resolved
#
# prepare the input for the function
# check and create valid _PermissionStruct parameter
Expand Down Expand Up @@ -254,6 +290,15 @@ def share(self, permission: Permission = None, shared_prefix: [SharePrefix] = No
#
# if error occurred
if bool(access_result.error):
raise _storj_exception(access_result.error.contents.code,
access_result.error.contents.message.decode("utf-8"))
return Access(access_result.access, self.uplink)
error_code = access_result.error.contents.code
error_msg = access_result.error.contents.message.decode("utf-8")

self.uplink.m_libuplink.uplink_free_access_result(access_result)

raise _storj_exception(error_code,error_msg)

access = Access(access_result.access, self.uplink)

self.uplink.m_libuplink.uplink_free_access_result(access_result)

return access
47 changes: 33 additions & 14 deletions uplink_python/download.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ def read(self, size_to_read: int):
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t]
self.uplink.m_libuplink.uplink_download_read.restype = _ReadResult
self.uplink.m_libuplink.uplink_free_read_result.argstypes = [_ReadResult]
Fadila82 marked this conversation as resolved.
Show resolved Hide resolved
#
# prepare the inputs for the function
data_size = ctypes.c_int32(size_to_read)
Expand All @@ -85,18 +86,25 @@ def read(self, size_to_read: int):
size_to_read)
#
# if error occurred
if bool(read_result.error):
raise _storj_exception(read_result.error.contents.code,
read_result.error.contents.message.decode("utf-8"))
if read_result.error:
error_code = read_result.error.contents.code
error_msg = read_result.error.contents.message.decode("utf-8")
self.uplink.m_libuplink.uplink_free_read_result(read_result)

raise _storj_exception(error_code,error_msg)

bytes_read = int(read_result.bytes_read)
data_read = bytes()
if int(read_result.bytes_read) != 0:
if bytes_read != 0:
#
# --------------------------------------------
# data conversion to type python readable form
# conversion of LP_c_ubyte to python readable data variable
data_read = ctypes.string_at(data_to_write_ptr, int(read_result.bytes_read))
return data_read, int(read_result.bytes_read)

self.uplink.m_libuplink.uplink_free_read_result(read_result)

return data_read, bytes_read

def read_file(self, file_handle, buffer_size: int = 0):
"""
Expand All @@ -120,8 +128,7 @@ def read_file(self, file_handle, buffer_size: int = 0):
if not buffer_size:
buffer_size = COPY_BUFSIZE
file_size = self.file_size()
if buffer_size > file_size:
buffer_size = file_size
buffer_size = min(buffer_size, file_size)
while file_size:
buf, bytes_read = self.read(buffer_size)
if buf:
Expand All @@ -141,14 +148,19 @@ def file_size(self):
self.uplink.m_libuplink.uplink_stat_object.argtypes = [ctypes.POINTER(_ProjectStruct),
ctypes.c_char_p, ctypes.c_char_p]
self.uplink.m_libuplink.uplink_stat_object.restype = _ObjectResult
self.uplink.m_libuplink.uplink_free_object_result.argtypes = [_ObjectResult]
#
# get object information by calling the exported golang function
object_result = self.uplink.m_libuplink.uplink_stat_object(self.project, self.bucket_name,
self.storj_path)
# if error occurred
if bool(object_result.error):
raise _storj_exception(object_result.error.contents.code,
object_result.error.contents.message.decode("utf-8"))
error_code = object_result.error.contents.code
error_msg = object_result.error.contents.message.decode("utf-8")

self.uplink.m_libuplink.uplink_free_object_result(object_result)

raise _storj_exception(error_code, error_msg)
# find object size
return int(object_result.object.contents.system.content_length)

Expand All @@ -170,8 +182,7 @@ def close(self):
#
# if error occurred
if bool(error):
raise _storj_exception(error.contents.code,
error.contents.message.decode("utf-8"))
self.uplink.free_error_and_raise_exception(error)

def info(self):
"""
Expand All @@ -191,6 +202,14 @@ def info(self):
#
# if error occurred
if bool(object_result.error):
raise _storj_exception(object_result.error.contents.code,
object_result.error.contents.message.decode("utf-8"))
return self.uplink.object_from_result(object_result.object)
error_code = object_result.error.contents.code
error_msg = object_result.error.contents.message.decode("utf-8")

self.uplink.m_libuplink.uplink_free_object_result(object_result)

raise _storj_exception(error_code, error_msg)

_object = self.uplink.object_from_result(object_result.object)
self.uplink.m_libuplink.uplink_free_object_result(object_result)

return _object
6 changes: 3 additions & 3 deletions uplink_python/hello_storj.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@

from datetime import datetime

from .errors import StorjException, BucketNotEmptyError, BucketNotFoundError
from .module_classes import ListObjectsOptions, Permission, SharePrefix
from .uplink import Uplink
from uplink_python.errors import StorjException, BucketNotEmptyError, BucketNotFoundError
from uplink_python.module_classes import ListObjectsOptions, Permission, SharePrefix
from uplink_python.uplink import Uplink

if __name__ == "__main__":

Expand Down
Loading