Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialization go #261

Open
wants to merge 177 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
177 commits
Select commit Hold shift + click to select a range
de0938e
Added --read-write switch
GreyCat Mar 24, 2017
d24a36d
JavaMain: ensure that exceptions are really thrown, even on compile s…
GreyCat Mar 24, 2017
0b757b9
Started PoC seq writer support in JavaCompiler
GreyCat Mar 24, 2017
93b37a1
JavaCompiler: very basic bytes writing support
GreyCat Mar 24, 2017
75ec0c2
Allow simple serialization of FixedBytesType
GreyCat Mar 24, 2017
eed5d07
Translators: added strToBytes, implemented in Java translator
GreyCat Mar 24, 2017
5cf4edd
Added _i iteration number identifier
GreyCat Mar 24, 2017
4a3535f
JavaTranslator: added translation of `_i`
GreyCat Mar 24, 2017
568964d
ClassTypeProvider: use constants + added ITERATOR_I handling
GreyCat Mar 24, 2017
9e7bffa
Writing: added string handling, some repeat-expr handling, BytesEos h…
GreyCat Mar 24, 2017
8fd1002
Serialization: basic support for user types and BytesLimitType
GreyCat Mar 27, 2017
0d73de2
Merge branch 'master' into serialization
GreyCat Mar 27, 2017
ea06fec
Serialization: implemented unprocess and slightly better user type wr…
GreyCat Mar 27, 2017
9c1793e
AllocateIOLocalVar: Added allocation of fixed and growing IOs
GreyCat Mar 28, 2017
9f0e331
Serialization: implemented fixed-size preallocated IO buffers
GreyCat Mar 28, 2017
969401c
JavaCompiler: generate new Readable/Writable interface implements
GreyCat Mar 28, 2017
885469f
Serialization: more intricate process-on-top-of-user-type support
GreyCat Mar 28, 2017
0f3a2aa
JavaCompiler: implemented writing of repeat-eos
GreyCat Mar 29, 2017
e4a56f1
Serialization: implemented switch types support, reworked JavaCompile…
GreyCat Mar 29, 2017
44185e1
Added basic _check implementation, added checks for a proper number o…
GreyCat Mar 30, 2017
769dabc
Added String#to_b(encoding) and ByteArray#size methods + JavaTranslat…
GreyCat Mar 30, 2017
040b689
Added limited byte / string sizing checks
GreyCat Mar 30, 2017
28e8d06
Translators: added BytesType#first and #last, implemented for Java
GreyCat Mar 30, 2017
1b66897
GenericChecks: implemented tons of checks for BytesLimitType - bounda…
GreyCat Mar 30, 2017
7d2295f
Merge branch 'master' into serialization
GreyCat Apr 3, 2017
dd0880e
Merge branch 'master' into serialization
GreyCat Apr 4, 2017
d73ec6b
JavaCompiler: only use ReadOnly when _read is public, i.e. when debug…
GreyCat Apr 5, 2017
2081fb2
Merge branch 'master' into serialization
GreyCat Apr 7, 2017
7dee284
Merge branch 'master' into serialization
GreyCat Apr 27, 2017
75729f7
WIP
generalmimon Oct 27, 2019
6e6bdda
Merge branch 'master' into serialization
generalmimon Nov 3, 2019
9eb2ab5
Merge pull request #177 from generalmimon/serialization
GreyCat Nov 10, 2019
2e2e985
Stop generating empty constructor when autoRead
generalmimon Nov 16, 2019
e46cc95
Use common loop for all repetitions
generalmimon Nov 16, 2019
9d8f2d3
Merge pull request #182 from generalmimon/serialization-common-loop
GreyCat Dec 14, 2019
e4ad9aa
Revert "Stop generating empty constructor when autoRead"
generalmimon Dec 14, 2019
d8adcc5
Disable autoRead automatically in readWrite mode
generalmimon Feb 2, 2020
cfba0cc
Merge pull request #180 from generalmimon/serialization-autoread-empt…
GreyCat Feb 3, 2020
d26e37b
Extract changes from https://github.com/kaitai-io/kaitai_struct_compi…
generalmimon Sep 19, 2020
ead02a4
Merge tag '0.10' into serialization
generalmimon Aug 1, 2022
5c70d8f
Fix build errors after upgrading to the 0.10 codebase
generalmimon Aug 1, 2022
0867d51
Remove `FixedBytesType` as intended in the 0.9 `valid` design
generalmimon Aug 16, 2022
939cf98
Refactor EveryWriteIsExpression to use Ast.expr.InternalName
generalmimon Aug 28, 2022
83b1fa7
Delete unused attrFixedContentsParse (_io.ensure_fixed_contents())
generalmimon Sep 4, 2022
caea2b7
Fix writing of a field with `terminator` + `consume: false`
generalmimon Sep 4, 2022
c1d391a
Fix broken consistency check for repeated byte arrays
generalmimon Sep 5, 2022
43d044a
Java: fix bytesSubscript to give unsigned byte values (finally!)
generalmimon Sep 5, 2022
8404319
Drop `expr` argument of condIfFooter method
generalmimon Sep 8, 2022
2acf924
Java: include term if `include: true` && `terminator` == `pad-right`
generalmimon Sep 8, 2022
73b0347
Fix & add missing `terminator`/`pad-right` consistency checks
generalmimon Sep 10, 2022
d721d08
Reuse the byte array write method for strings
generalmimon Sep 11, 2022
50d6556
Add repeat-until last element check
generalmimon Sep 12, 2022
72ca925
Refactor access to the last array item in attrAssertUntilCond
generalmimon Sep 12, 2022
426a38d
Merge pull request #183 from generalmimon/serialization-repeat-until-…
generalmimon Sep 12, 2022
def6650
Extend repeat-until consistency check to all elements
generalmimon Sep 12, 2022
306bfb2
Add consistency check for *non-empty* `repeat: until` array
generalmimon Sep 12, 2022
849fc1c
Fix "Object cannot be converted to byte[]" errors in type switches
generalmimon Sep 13, 2022
74d553c
Fix locale-sensitive String#toUpperCase call in writeHeader
generalmimon Sep 14, 2022
604359d
Java: finally solve boxed numeric type casting issues consistently
generalmimon Sep 15, 2022
f8bb577
Add _invalidate*() for value and set*() for parse instances
generalmimon Sep 17, 2022
16374cc
Add _write*() and _check*() for parse instances
generalmimon Sep 17, 2022
070391c
Set write flags of parse instances at the beginning of `_write`
generalmimon Oct 28, 2022
9897818
_write(): use special loops for each repetition type again
generalmimon Oct 28, 2022
69e4a12
Java: make {_read,_write}{BE,LE}() `private` even if autoRead is off
generalmimon Oct 28, 2022
2710191
Fix instance write flags (previous attempt led to infinite recursion)
generalmimon Oct 28, 2022
7dce0c3
Rename _write{,LE,BE}() methods to _write_Seq{,LE,BE}()
generalmimon Oct 28, 2022
826b242
Rename {writeExprAsExpr=>itemExpr}(), move to EveryReadIsExpression
generalmimon Oct 29, 2022
ca012b6
Add _fetchInstances() method to recursively fetch parse instances
generalmimon Oct 29, 2022
3b08efe
Write substreams to parent only after all fields have been written
generalmimon Nov 26, 2022
544cad0
Add set{Inst}_ToWrite() setter to select what instances to write
generalmimon Nov 26, 2022
11da92f
Add support for writing substreams with `process` and/or `terminator`
generalmimon Dec 23, 2022
175c278
Fix handling `size-eos: true` with `terminator`/`pad-right`
generalmimon Dec 23, 2022
b5c2e92
Java: call writeBitsInt{Be,Le}(), disable calls to alignToByte()
generalmimon Dec 28, 2022
bda1842
Move consistency checks from _check() to _write() if needed
generalmimon Dec 31, 2022
a5dd1f9
Add a _write() check on EOF after a `size-eos: true` field
generalmimon Dec 31, 2022
ecb5757
Add a _write() check on EOF states of `repeat: eos`
generalmimon Dec 31, 2022
37f432b
Generate byte array checks for substreams in _write()
generalmimon Dec 31, 2022
4fdf21d
Add write support for `eos-error: false`
generalmimon Jan 1, 2023
a8e4c03
Convert Identifier to id in ConsistencyError using `humanReadable`
generalmimon Jan 1, 2023
08ac809
Remove unused `isRaw` parameter in `itemExpr` method
generalmimon Jan 1, 2023
cf26f3e
Generate checks for SwitchType as well
generalmimon Jan 1, 2023
37da6dd
Add checks for user `params`, set `io`/`io[]` params in _write()
generalmimon Jan 1, 2023
2accf38
Add checks for `_root` and `_parent` built-in parameters
generalmimon Jan 2, 2023
5afa962
Add `valid` checks as in _read() to _check()/_write()
generalmimon Jan 2, 2023
793f58b
Add necessary fixes after testing on https://github.com/kaitai-io/kai…
generalmimon Jan 3, 2023
533aaa8
Going forward, starting 0.11-SNAPSHOT
GreyCat Aug 1, 2022
52812f9
Fix superfluous footer in Lua, Perl, PHP, Python and Ruby
generalmimon Feb 14, 2023
203613b
Delete unused method attrWriteStreamToStream
generalmimon Feb 15, 2023
d1f16dd
Port serialization support to Python
generalmimon Feb 26, 2023
d986731
Fix Python 2 compatibility when creating a fixed stream
generalmimon Feb 26, 2023
fca43d9
Python: disable calls to alignToByte()
generalmimon Mar 6, 2023
e776c98
Python: use `self` as default `_root` only in top-level types
generalmimon Mar 7, 2023
00c831f
Add "implies --no-auto-read" to --read-write CLI option description
generalmimon Jul 27, 2023
f1dc857
Add "Java and Python only" to --read-write CLI option description
generalmimon Jul 27, 2023
952c89c
Merge branch 'master' into serialization
generalmimon Jul 31, 2023
9142a1b
Improve setting `zeroCopySubstream` to false in `readWrite` mode
generalmimon Aug 1, 2023
2eca3de
Bring back init of instance flags in non-`readWrite` mode (C++, C#)
generalmimon Aug 1, 2023
a477800
PythonCompiler: fix "{alignToByte => align_to_byte}()" in comment
generalmimon Aug 5, 2023
8b258e2
Remove alignToByte() insertion logic when writing
generalmimon Aug 5, 2023
cb0c1eb
Endianness.fromString(): revert incorrect `case {None => _}` change
generalmimon Aug 7, 2023
0179199
Stop using `_io` when throwing validation errors from _check()
generalmimon Sep 24, 2023
31f1359
Merge branch 'master' into serialization
generalmimon Oct 12, 2023
7d830fa
Fix Java 7 compat: add `final` in allocateIOFixed()
generalmimon Oct 13, 2023
dea74e1
Merge branch 'master' into serialization
generalmimon Oct 21, 2023
aed4570
feat: go translator add strToBytes
Nov 22, 2023
72329c9
TMP1
Nov 23, 2023
a267d7e
TMP2
Nov 23, 2023
cdacc80
TMP3
Nov 24, 2023
d52c49d
fix: fixup type generation
Nov 28, 2023
6c2c4aa
fix
Nov 28, 2023
a11ba8b
Merge pull request #2 from yywing/fix/fix-type-generate
yywing Nov 29, 2023
64dfdf6
feat: draft support go write generate
Nov 30, 2023
1489385
Merge pull request #3 from yywing/feat/add-go-write
yywing Dec 1, 2023
966c3f5
fix: fix type generate and class generaye
Dec 1, 2023
6d09fa2
feat: fix go write generate and type check with err check problem
Dec 4, 2023
6c687b9
fix: fix interface{} type generate
Dec 5, 2023
ac53133
fix: fixup interface{} and int type convert
Dec 6, 2023
5e962a7
fix: add return res value
Dec 7, 2023
89ef039
fix: fix default case type transport
Dec 7, 2023
76f8ce7
del
Dec 7, 2023
2f518bf
fix: fix types
Dec 7, 2023
993f204
fix: fix type convert in kaitai-struct-test
Dec 8, 2023
cd5eb73
fix: fix Write
Dec 8, 2023
3e5bdb5
fix: fix invalid instatnce type
Dec 8, 2023
fe4c092
fix: fix invalid types
Dec 11, 2023
2d0b6f6
fix: fix type and add attr setter
Dec 11, 2023
fa02c4c
fix: fix child with wrong io
Dec 11, 2023
ac461c4
fix
Dec 11, 2023
5b703df
fix: set invalid_ switch to false
Dec 11, 2023
265dcb1
fix: delete repeat eof check and fix type
Dec 12, 2023
5bdf4fb
feat: add size calc
Dec 12, 2023
429a2ba
feat: support go enum with type declare
Dec 12, 2023
08f4a30
fix: split boundary
Dec 12, 2023
5f1eda6
fix: wrong type convert
Dec 12, 2023
65dd7cf
fix
Dec 12, 2023
d834f4a
fix: temp fixup terminated type support
Dec 12, 2023
edb6d1d
fix: add more convert to bytes
Dec 12, 2023
93c7735
fix: fixup terminated type generate
Dec 13, 2023
ff22449
fix: fix bytes type
Dec 13, 2023
7c9f62b
fix: changto write
Dec 13, 2023
9219a36
fix: fixup write
Dec 14, 2023
17a427a
fix: fixup some type convert
Dec 14, 2023
327a811
fix: fix terminated check
Dec 15, 2023
e143fa1
fix: fix binop type transform
Dec 15, 2023
9489f80
fix: delete array bracket
Dec 18, 2023
8208c40
fix: fix some types
Dec 18, 2023
f37238c
fix: delete *
Dec 18, 2023
14e6256
fix: interface tmp(WIP) and calc expr convert
Dec 19, 2023
6d56a31
fix: interface{} type convert and some bug fix
Dec 20, 2023
22130aa
fix: ascii encoding
Dec 22, 2023
ad357e8
fix: read default
Dec 22, 2023
708779e
fix: map get value
Dec 22, 2023
f83cfc7
fix: delete unuseful code
Dec 22, 2023
c9b8f62
fix: clean up unuseful code generate and fix up interface type's conv…
Dec 25, 2023
301aa24
fix: WriteBytesLimit wrong size type
Dec 26, 2023
ba55982
fix: check with wrong value
Dec 26, 2023
fdd946f
fix: go len type compare
Dec 27, 2023
fcee2a2
fix: rune call with wrong string type
Jan 3, 2024
0280569
fix: resolve seq child has more than one parent but type is kaitaistruct
Jan 5, 2024
9480d5a
draft: fix: add calctype for switch case interface
Jan 10, 2024
d603eeb
fix: bit shift type convert
Jan 15, 2024
e27f498
fix: add parent nil detect
Jan 15, 2024
58b97e2
fix
Jan 16, 2024
ebaa561
fix: parseint type transform
Jan 16, 2024
26d403c
fix: golang writeseq + meta switch-on endian => no endian change
Jul 2, 2024
6399fd5
fix: unused case
Jul 3, 2024
93464b9
fix: iter assignment
Jul 4, 2024
c2e66bc
fix subscribe type with wrong type convert
Jul 5, 2024
c419dba
fix: move struct pointer to front
Jul 10, 2024
59dbb65
fix: generate
Oct 14, 2024
cd2d8d0
fix
Oct 14, 2024
d819f8f
fix: root_struct with root self-signed
Dec 5, 2024
87f2023
fix: kaitai size calc
Dec 17, 2024
65c3477
fix
Dec 17, 2024
7cb15b7
fix: case
Dec 18, 2024
4b617ad
fix: bit_1 -> bool in go -> 1 bytes
Dec 18, 2024
c14ed7e
fix
Dec 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .DS_Store
Binary file not shown.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,6 @@ project/boot/
# Unpacking complete binaries for testing purposes
/runnable
js/npm

.bsp/
.metals/
6 changes: 4 additions & 2 deletions .idea/runConfigurations/ksc_formats_all.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"files.watcherExclude": {
"**/target": true
}
}
85 changes: 63 additions & 22 deletions jvm/src/main/scala/io/kaitai/struct/JavaMain.scala
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import io.kaitai.struct.languages.CppCompiler
import io.kaitai.struct.languages.components.LanguageCompilerStatic
import io.kaitai.struct.problems.{CompilationProblem, CompilationProblemException, ErrorInInput}

import scala.collection.immutable.Map

object JavaMain {
KSVersion.current = Version.version

Expand Down Expand Up @@ -57,6 +59,10 @@ object JavaMain {
}
}

opt[Unit]('w', "read-write") action { (x, c) =>
c.copy(runtime = c.runtime.copy(readWrite = true, autoRead = false))
} text("generate read-write support in classes (implies `--no-auto-read --zero-copy-substream false`, Java and Python only, default: read-only)")

opt[File]('d', "outdir") valueName("<directory>") action { (x, c) =>
c.copy(outDir = x)
} text("output directory (filenames will be auto-generated); on Unix-like shells, the short form `-d` requires arguments to be preceded by `--`")
Expand Down Expand Up @@ -172,7 +178,13 @@ object JavaMain {
version("version") text("output version information and exit")
}

parser.parse(args, CLIConfig())
parser.parse(args, CLIConfig()).map { c =>
if (c.runtime.readWrite) {
c.copy(runtime = c.runtime.copy(zeroCopySubstream = false))
} else {
c
}
}
}

/**
Expand Down Expand Up @@ -259,8 +271,8 @@ class JavaMain(config: CLIConfig) {
def run(): Unit = {
val logs: Map[String, InputEntry] = config.srcFiles.map { srcFile =>
val log = if (config.throwExceptions) {
compileOneInput(srcFile)
} else {
compileOneInput(srcFile) } else {

try {
compileOneInput(srcFile)
} catch {
Expand Down Expand Up @@ -292,26 +304,55 @@ class JavaMain(config: CLIConfig) {

private def compileOneInput(srcFile: String): InputEntry = {
Log.fileOps.info(() => s"parsing $srcFile...")
val (specsOpt, precompileProblems) = JavaKSYParser.localFileToSpecs(srcFile, config)

specsOpt match {
case Some(specs) =>
val output: Map[String, Map[String, SpecEntry]] = config.targets match {
case Seq(lang) =>
// single target, just use target directory as is
val out = compileOneLang(specs, lang, config.outDir.toString)
Map(lang -> out)
case _ =>
// multiple targets, use additional directories
compileAllLangs(specs, config)

val dir = new File(srcFile)

if ( dir.isDirectory ) {
var recordInput: InputSuccess = null
for ( f: File <- dir.listFiles() ) {
val (specsOpt, precompileProblems) = JavaKSYParser.localFileToSpecs(f.getPath, config)
specsOpt match {
case Some(specs) =>
val output: Map[String, Map[String, SpecEntry]] = config.targets match {
case Seq(lang) =>
// single target, just use target directory as is
val out = compileOneLang(specs, lang, config.outDir.toString)
Map(lang -> out)
case _ =>
// multiple targets, use additional directories
compileAllLangs(specs, config)
}
recordInput = InputSuccess(
specs.firstSpec.nameAsStr,
output,
precompileProblems
)
case None =>
InputFailure(precompileProblems)
}
InputSuccess(
specs.firstSpec.nameAsStr,
output,
precompileProblems
)
case None =>
InputFailure(precompileProblems)
}
recordInput
} else {
val (specsOpt, precompileProblems) = JavaKSYParser.localFileToSpecs(srcFile, config)
specsOpt match {
case Some(specs) =>
val output: Map[String, Map[String, SpecEntry]] = config.targets match {
case Seq(lang) =>
// single target, just use target directory as is
val out = compileOneLang(specs, lang, config.outDir.toString)
Map(lang -> out)
case _ =>
// multiple targets, use additional directories
compileAllLangs(specs, config)
}
InputSuccess(
specs.firstSpec.nameAsStr,
output,
precompileProblems
)
case None =>
InputFailure(precompileProblems)
}
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -692,8 +692,8 @@ class TranslatorSpec extends AnyFunSuite {
val langs = Map[LanguageCompilerStatic, AbstractTranslator with TypeDetector](
CppCompiler -> new CppTranslator(tp, new CppImportList(), new CppImportList(), RuntimeConfig()),
CSharpCompiler -> new CSharpTranslator(tp, new ImportList()),
GoCompiler -> new GoTranslator(goOutput, tp, new ImportList()),
JavaCompiler -> new JavaTranslator(tp, new ImportList()),
GoCompiler -> new GoTranslator(goOutput, tp, new ImportList(), RuntimeConfig()),
JavaCompiler -> new JavaTranslator(tp, new ImportList(), RuntimeConfig()),
JavaScriptCompiler -> new JavaScriptTranslator(tp),
LuaCompiler -> new LuaTranslator(tp, new ImportList()),
PerlCompiler -> new PerlTranslator(tp, new ImportList()),
Expand Down
23 changes: 23 additions & 0 deletions project/.bloop/bloop.settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"javaSemanticDBVersion": "0.9.6",
"semanticDBVersion": "4.8.12",
"supportedScalaVersions": [
"2.13.12",
"2.12.18",
"2.12.17",
"2.12.16",
"2.12.15",
"2.13.9",
"2.13.10",
"2.13.11",
"2.11.12",
"2.12.11",
"2.12.12",
"2.12.13",
"2.12.14",
"2.13.5",
"2.13.6",
"2.13.7",
"2.13.8"
]
}
23 changes: 23 additions & 0 deletions project/project/.bloop/bloop.settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"javaSemanticDBVersion": "0.9.6",
"semanticDBVersion": "4.8.12",
"supportedScalaVersions": [
"2.13.12",
"2.12.18",
"2.12.17",
"2.12.16",
"2.12.15",
"2.13.9",
"2.13.10",
"2.13.11",
"2.11.12",
"2.12.11",
"2.12.12",
"2.12.13",
"2.12.14",
"2.13.5",
"2.13.6",
"2.13.7",
"2.13.8"
]
}
Loading