Skip to content
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rerun the scripts, these file are not generated anymore

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified weights_tf_free.h5 is up to date

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script generates weights.h5. It does not generate weights_tf_free.h5 or weights_tf_free.npz.

Copy link
Author

@anushka-cseatmnc anushka-cseatmnc Mar 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image it does kindly review made changes again.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete these files and regenerate them

Copy link
Author

@anushka-cseatmnc anushka-cseatmnc Mar 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@robertbastian I have deleted the outdated weights_tf_free.h5 as requested and regenerated the necessary files. The new weights_tf_free.h5 files are now up-to-date. Kindly review the changes again. here is screenshot i'm attaching for reference - If there is any more changes required kindly let me know .
Screenshot 2025-03-17 225025

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this is not the output of the save_model method. In fact, you have reverted the changes to that method.

Copy link
Author

@anushka-cseatmnc anushka-cseatmnc Mar 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you give me insights what I'm exactly supposed to do??

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2025-03-17 225043
is this what you r asking about ?

Copy link
Author

@anushka-cseatmnc anushka-cseatmnc Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sffc @robertbastian please review changes . And guidence will be helpful for further changes.

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
48 changes: 48 additions & 0 deletions convert_weights.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import os
import numpy as np
import tensorflow as tf # Needed to handle TF tensors
import h5py

def convert_to_numpy(value):
"""
Convert TensorFlow tensors/variables to NumPy arrays (float32).
Ensures we remove any TensorFlow-specific data.
"""
if isinstance(value, tf.Tensor) or isinstance(value, tf.Variable):
return value.numpy().astype(np.float32)
elif isinstance(value, np.ndarray) and np.issubdtype(value.dtype, np.number):
return value.astype(np.float32)
else:
return None # Ignore non-numeric data

def convert_weights(npy_path):
"""Convert `weights.npy` to a TensorFlow-free HDF5 format."""
if not os.path.exists(npy_path):
print(f"❌ Error: {npy_path} not found!")
return

h5_path = npy_path.replace(".npy", "_tf_free.h5")

# Load the weights
print(f"πŸ” Loading {npy_path}...")
weights = np.load(npy_path, allow_pickle=True)

# Convert all elements to NumPy arrays (remove TensorFlow dtypes)
converted_weights = [convert_to_numpy(w) for w in weights if convert_to_numpy(w) is not None]

# Save to HDF5 format
with h5py.File(h5_path, "w") as hf:
for i, w in enumerate(converted_weights):
hf.create_dataset(f"weight_{i}", data=w)

print(f"βœ… Converted: {npy_path} -> {h5_path}")

if __name__ == "__main__":
# Search for all `weights.npy` files and convert them
for root, _, files in os.walk("."):
for file in files:
if file == "weights.npy":
convert_weights(os.path.join(root, file))

print("πŸš€ All weight files converted successfully!")

1,414 changes: 710 additions & 704 deletions lstm_word_segmentation/word_segmenter.py

Large diffs are not rendered by default.