Skip to content

Commit

Permalink
fix corner case of small traces
Browse files Browse the repository at this point in the history
  • Loading branch information
rayandrew committed Oct 4, 2024
1 parent 21ff5e3 commit a0c0835
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 9 deletions.
11 changes: 5 additions & 6 deletions docs/utilities.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
========================
DFTracer Utility scripts
DFTracer Utility scripts
========================

This section describes the utilities provided by DFTracer to assist users with logs.
Expand All @@ -12,14 +12,14 @@ All scripts are installed with DFTracer in the installation's directories bin fo
Merge Trace script
------------------

This script allows users to combine all pfw format into one.
This script allows users to combine all pfw format into one.
This has the following signature.

.. code-block:: bash
<install-dir>/bin/dftracer_merge [-fcv] [-d input_directory] [-o output_file]
Arguments for this script are
Arguments for this script are

1. **-d input_directory** folder containing all trace files. Default `PWD`.
2. **-o output_file** file for storing merged file. Default `combined.pfw`.
Expand All @@ -38,7 +38,7 @@ The script compacts all trace file and then divides the trace into equal file pi
<install-dir>/bin/dftracer_compact [-fcv] [-d input_directory] [-o output_directory] [-l num_lines] [-p prefix]
Arguments for this script are
Arguments for this script are

1. **-d input_directory** specify input directories. Should contain .pfw or .pfw.gz files. Default `PWD`.
2. **-o output_file** specify output directory. Default `combined.pfw`.
Expand All @@ -49,7 +49,6 @@ Arguments for this script are
7. **-v** enable verbose mode.
8. **-h** display help


------------------
Sanitize script
------------------
Expand Down Expand Up @@ -110,7 +109,7 @@ Arguments for this script are:
Splitting DFTracer traces
------------------

The script splits the traces into equal sized chunk optiimzed for analysis
The script splits the traces into equal sized chunk optimized for analysis

.. code-block:: bash
Expand Down
12 changes: 9 additions & 3 deletions script/dftracer_split
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash

# The script splits the traces into equal sized chunk optiimzed for analysis
# The script splits the traces into equal sized chunk optimized for analysis
# This has the following signature.
#
# usage: dftracer_split [-fv] [-n app_name] [-d input_directory] [-o output_directory] [-s chunk_size]
Expand Down Expand Up @@ -172,15 +172,21 @@ for file in sorted(glob.glob("*.pfw.gz")):
accumulated_size += file_size_mb
CHUNKS.append({"path": file, "size": file_size_mb, "start": start + 1, "end": max_lines + 1})
file_size_mb = 0
if accumulated_size > 0:
print(json.dumps({"chunks": CHUNKS, "size": accumulated_size}))
if len(CHUNKS) > 0:
print(json.dumps({"chunks": CHUNKS, "size": accumulated_size}))
accumulated_size = 0.0
CHUNKS = []
EOF)
i=0
# count number of chunks by newline
total_chunks=$(wc -l <<< "$PY_OUT")
total_chunks=$((total_chunks - 1))
JOBS_LIMIT=$(nproc --all)
# echo $PY_OUT
# echo $total_chunks
for line in $PY_OUT; do
running_jobs=$(jobs -rp | wc -l)
if [ $running_jobs -ge $JOBS_LIMIT ]; then
Expand Down

0 comments on commit a0c0835

Please sign in to comment.