realKfiros#269
Open
realKfiros wants to merge 22 commits into
Open
Conversation
…prove CSV parsing logic.
…aration and restructuring output logic.
…teJson` method for improved clarity and reusability.
…roperties and modularizing visit count, grouping, and JSON writing into dedicated methods.
…h a direct `file_put_contents` call.
…tion and JSON output generation.
…on by introducing chunk-based processing, path discovery, and modularized methods.
… improved efficiency and modularity, leveraging `Visit::all()` for path retrieval.
… caching, streamline line handling, and simplify visit aggregation logic.
…ED_SLASHES` for improved JSON readability.
Author
|
/bench |
Member
|
Benchmarking failed: Parsed result did not match expected result |
…json so they would be in the order of the csv and not alphanetically, change to 4 spaces instead of tabs and remove the last line
Author
|
/bench |
Member
|
Benchmarking complete! Best execution time: 36.097s Full results: {
"results": [
{
"command": "cd /Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../.benchmark/pr-269 && php -dmax_execution_time=300 tempest data:parse --input-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data.csv\" --output-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data-actual.json\"",
"mean": 36.09780654822,
"stddev": null,
"median": 36.09780654822,
"user": 34.326457919999996,
"system": 1.75124006,
"min": 36.09780654822,
"max": 36.09780654822,
"times": [
36.09780654822
],
"memory_usage_byte": [
60473344
],
"exit_codes": [
0
]
}
]
}
|
…timize visit counting, and modularize path discovery logic.
Author
|
/bench |
Member
|
Benchmarking failed: Parsed result did not match expected result |
Member
|
Benchmarking complete! Best execution time: 64.632s Full results: {
"results": [
{
"command": "cd /Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../.benchmark/pr-269 && php -dmax_execution_time=300 tempest data:parse --input-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data.csv\" --output-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data-actual.json\"",
"mean": 64.63219923247999,
"stddev": null,
"median": 64.63219923247999,
"user": 62.94510572,
"system": 1.6843587199999999,
"min": 64.63219923247999,
"max": 64.63219923247999,
"times": [
64.63219923247999
],
"memory_usage_byte": [
59408384
],
"exit_codes": [
0
]
}
]
}
|
…tions and refining path and visit counting logic.
Author
|
/bench |
Member
|
Benchmarking complete! Best execution time: 46.665s Full results: {
"results": [
{
"command": "cd /Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../.benchmark/pr-269 && php -dmax_execution_time=300 tempest data:parse --input-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data.csv\" --output-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data-actual.json\"",
"mean": 46.66587695188,
"stddev": null,
"median": 46.66587695188,
"user": 45.024278939999995,
"system": 1.6388568399999999,
"min": 46.66587695188,
"max": 46.66587695188,
"times": [
46.66587695188
],
"memory_usage_byte": [
59932672
],
"exit_codes": [
0
]
}
]
}
|
…, optimizing visit counting, and improving JSON output to maintain CSV order.
Author
|
/bench |
…p`, optimize visit counting with cached path-base lookup, and improve error handling during file processing.
Member
|
Benchmarking complete! Best execution time: 16.305s Full results: {
"results": [
{
"command": "cd /Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../.benchmark/pr-269 && php -dmax_execution_time=300 tempest data:parse --input-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data.csv\" --output-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data-actual.json\"",
"mean": 16.305720439,
"stddev": null,
"median": 16.305720439,
"user": 15.635367259999999,
"system": 0.66457596,
"min": 16.305720439,
"max": 16.305720439,
"times": [
16.305720439
],
"memory_usage_byte": [
59637760
],
"exit_codes": [
0
]
}
]
}
|
Member
|
I think there's room for one more improvement… 👀 |
Member
|
You didn’t optimize. You performed violence (on latency). 🔪 |
… simplifying visit counting with cached maps, and improving JSON output generation.
Author
|
/bench |
Member
|
Benchmarking complete! Best execution time: 16.191s Full results: {
"results": [
{
"command": "cd /Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../.benchmark/pr-269 && php -dmax_execution_time=300 tempest data:parse --input-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data.csv\" --output-path=\"/Users/brentroose/Dev/100-million-row-challenge/app/Commands/../../data/real-data-actual.json\"",
"mean": 16.19115328166,
"stddev": null,
"median": 16.19115328166,
"user": 15.53599716,
"system": 0.65215882,
"min": 16.19115328166,
"max": 16.19115328166,
"times": [
16.19115328166
],
"memory_usage_byte": [
59817984
],
"exit_codes": [
0
]
}
]
}
|
Member
|
Mean time goes down, confidence goes up |
Member
|
You shaved off time so clean it should be in a barbershop. 💈 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This submission is single threaded.