Sleigh - reuse table for different named fields?

[raybellis]

I'm trying to learn Sleigh to disassemble NS32016 code, and it's a hairy instruction set.

In particular there's a 5 bit field variously named gen, gen1 and gen2 that can appear in various (7?) different locations with varying length instructions, but with identical semantics in each case. Worse still, some values of that field can then require that additional tokens (of varying lengths) are read in from the instruction stream.

Is there a way in Sleigh to just create a single table that will work without it being specific to the particular bit placement of that field within the instruction?

Currently the code looks like this, but doesn't yet cover all of the possible addressing modes, and this is for just for one of the possible places that this field can appear:

gen: "R0" is fmt2_gen=0b00000 {}
gen: "R1" is fmt2_gen=0b00001 {}
gen: "R2" is fmt2_gen=0b00010 {}
gen: "R3" is fmt2_gen=0b00011 {}
gen: "R4" is fmt2_gen=0b00100 {}
gen: "R5" is fmt2_gen=0b00101 {}
gen: "R6" is fmt2_gen=0b00110 {}
gen: "R7" is fmt2_gen=0b00111 {}
gen: disp^"(R0)" is fmt2_gen=0b01000 ; disp {}
gen: disp^"(R1)" is fmt2_gen=0b01001 ; disp {}
gen: disp^"(R2)" is fmt2_gen=0b01010 ; disp {}
gen: disp^"(R3)" is fmt2_gen=0b01011 ; disp {}
gen: disp^"(R4)" is fmt2_gen=0b01100 ; disp {}
gen: disp^"(R5)" is fmt2_gen=0b01101 ; disp {}
gen: disp^"(R6)" is fmt2_gen=0b01110 ; disp {}
gen: disp^"(R7)" is fmt2_gen=0b01111 ; disp {}
gen: imm is fmt2_gen=0b10100; imm {}
gen: @disp is fmt2_gen=0b10101; disp {}
gen: "TOS" is fmt2_gen=0b10111 {}
gen: disp^("FP") is fmt2_gen=0b11000 ; disp {}
gen: disp^("SP") is fmt2_gen=0b11001 ; disp {}
gen: disp^("SB") is fmt2_gen=0b11010 ; disp {}
gen: * + disp is fmt2_gen=0b11011 ; disp {}

disp is a variable length field for signed displacements, identified by the top two bits in the first byte:

#
# Variable length displacements, big endian
#
define token disp8(8) endian=big
    d8_mask=(7,7)
    val8=(0,6) signed
;

define token disp16(16) endian=big
    d16_mask=(14,15)
    val16=(0,13) signed
;

define token disp32(32) endian=big
    d32_mask=(30,31)
    val32=(0,29) signed
;

#
# Displacement table
#
disp:  val8 is  d8_mask=0b0  &  val8 { export  val8; }
disp: val16 is d16_mask=0b10 & val16 { export val16; }
disp: val32 is d32_mask=0b11 & val32 { export val32; }

I tried using a context register as a way to "pass" a specific field instance that could be read in the disp table but the Sleigh compiler didn't allow that :( To make it worse some instructions can effectively end up with four gen type fields in them, where there are two primary gen fields each of which for values 0xb111XX can point to an index byte which itself contains a gen field!

Thanks for any advice!

[GhidorahRex]

I think your context register idea should work, although I don't know how you tried to execute it to know for sure.

There will always be some duplication with sleigh. It's just the nature of the code. But I think you only need to duplicate it 3 times. I looked through the NS32000 instruction set and all the gen operands occur in the first 2 5-bit segments of the first 10 bits, or in the index bytes.

For example:

!   src   !   dest  !           SCALBf          !
+---------+---------+---------+-+---------------+
!   gen   !   gen   !0 1 0 0 0!f!1 1 1 1 1 1 1 0!
!-+-+-+-+-+-+-+-!-+-+-+-+-+-+-+-!-+-+-+-+-+-+-+-!
23            16 15            8 7             0

or

!   src   !   dest  !    ORi    !
+---------+---------+-------+---+
!   gen   !   gen   !0 1 1 0! i !
!-+-+-+-+-+-+-+-!-+-+-+-+-+-+-+-!
 15            8 7             0 

You can write your tokens so that the first 16 bytes use the same token, and the SCALBf instruction uses an additional opcode token while the ORi one doesn't. So these two instructions are using the same two tables, something like SrcGen and DestGen.

Then you only have to worry about the additional index byte gen operand.

[raybellis]

Thanks - I'll try the context register approach again. If I do get that working, I still have a concern over whether I need to use two (or even four) such context registers, or will Sleigh just "do the right thing" when multiple gen fields' pcode successively update and/or access that state within a single instruction decode.

I'm puzzled by your suggestion of using the left-most 16 bits first as a token. The 32016 is little endian, so it's bits 0 - 7 that are getting read first. IIUC, the processor has to parse those bits first to determine the length of the base opcode, as either 8, 16 or 24 bits.

That said, my count of seven was probably wrong, looking again with I think it's five: two from the different places that gen appears in 16 bit instructions, another two in 24 bit instructions, and then one in an index byte.

(I have also managed to shrink the gen ruleset (for now) by using subfields and an attach for the three bit general register field when used)

Noting also that a way to re-use tables would be incredibly useful in other cases. On the 32016 as well as Bcond there's Scond, and currently that appears to require another 16 entry table to be replicated.

[raybellis]

Oh, I think I see now - although you wrote "first" you really just meant "most significant", so I could use the same 16 bit token to represent the "last" two bytes of both the 16-bit and 24-bit instructions.

[GhidorahRex]

Exactly. As far as the context register goes - you'll need a separate field for each gen since it would have to store them simultaneously, so I don't think it will save you all that much. I think you're best off doing 3 gen sub-table. But you can reuse the index tables.

[raybellis]

I've made some good progress, using the 16-bit token as suggested above even for 24-bit instructions, but now I'm having issues with a format 4 16-bit instruction with two gen operands where gen1=0b10100 (for an immediate constant extension operand) and gen2=0b10101 (a displacement extension operand):

The matching instruction and gen1 and gen2 table entries are:

:MOV^i  gen1,gen2 is (i & fmt4_op=0b0101)... & gen1 & gen2 {}

gen1: imm is i16_gen1=0b10100 ; imm { export imm; }
gen2: @disp is i16_gen2=0b10101 ; disp { export disp; }

with imm defined thus via a context register:

i: "B" is i16_i=0b00 [ isize = 0b00; ] {}
i: "W" is i16_i=0b01 [ isize = 0b01; ] {}
i: "D" is i16_i=0b11 [ isize = 0b11; ] {}

imm: "#"^imm8  is isize=0b00 &  imm8 { export *[const]:4 imm8; }
imm: "#"^imm16 is isize=0b01 & imm16 { export *[const]:4 imm16; }
imm: "#"^imm32 is isize=0b11 & imm32 { export *[const]:4 imm32; }

and where disp is as per my first message above.

For the instruction sequence 54 a5 03 c0 81 80 28 this should resolve to MOVB #3, @0x818028 but instead I get MOVB #3, @3, i.e. as if both ; imm and ; disp are reading the same 0x03 byte at offset 2 and only consuming one additional one-byte token instead of a one-byte immediate token followed by a displacement token which is variable length but that in this case would be four bytes long.

The instruction info is:

Instruction Summary
-------------------
Mnemonic          : MOVB
Number of Operands: 2
Address           : ram:00000065
Flow Type         : FALL_THROUGH
Fallthrough       : 00000068
Delay slot depth  : 0
Hash              : 7cfcaed4

Input Objects:
   const:0x3
Result Objects:
   
Constructor Line #'s:
   MOV(NS32016.slaspec:277), i(NS32016.slaspec:174), 
   gen1(NS32016.slaspec:211), imm(NS32016.slaspec:178), 
   gen2(NS32016.slaspec:224), disp2(NS32016.slaspec:133)

Byte Length : 3
Instr Bytes : 01010100 10100101 00000011
Mask        : 11111111 11111111 00000000
Masked Bytes: 01010100 10100101 00000000

Instr Context:
   [Instruction context has not been set]

I tried creating a second disp2 table and consuming that in the gen2 table instead but it made no difference.

Is there a good example in any of the existing processor Sleigh files for how to correctly consume the additional extension bytes (and in the correct order) ?

thanks!

p.s. if a separate question is preferred for search purposes let me know.

[raybellis]

I've tried that now but it seems to break the decode of the preceding MOVB instructions and leaves the disassembly window unresponsive.

The unidasm output of this code is:

0053: 94 a1 00                 MOVB    0x0, R6
0056: 54 35 c0 8e 40 00        MOVB    R6, @0x8E4000
005c: d4 a1 00                 MOVB    0x0, R7
005f: 54 3d c0 8e 40 02        MOVB    R7, @0x8E4002
0065: 54 a5 03 c0 81 80 28     MOVB    0x3, @0x818028

and Ghidra showing:

where disassembly breaks at 0x0056 whereas previously it was good up to (but not including) 0x0065.

[raybellis]

I did also try modifying the gen1 table for the register operand but it didn't help:

gen1: i16_gen1_r is (i16_gen1_op=0b00 & i16_gen1_r) ... { export i16_gen1_r; }

[GhidorahRex]

If you run the script DebugSleighInstructionParse it will show you what matched. Also the ReloadSleighLanguage script is very helpful during development, although it has trouble reparsing some things, but it's great for double-checking your operands like this.

It looks like it matched the wrong disp entry here, same at 0x005f. It shouldn't leave the window unresponsive though.

[raybellis]

With the disassembly window hung I can't run the DebugSleighInstructionParse script :(

I've uploaded my current code to https://github.com/raybellis/ghidra-NS32016.git . It's quite possible (indeed probable) that my entire approach is garbage, though! ;)

I've also barely attempted to populate any of the pcode sections yet.

[GhidorahRex]

I have no idea why the disassembly window is hung. I don't have your binary, but I can copy those bytes in and disassemble just fine (albeit incorrectly). Try restarting ghidra?

As far as rectifying the issue, it's complicated. Sleigh has trouble figuring out how many bytes should go in-between. You may need to include some skip bytes (again, see the M16C example, although I think that's overly complicated), which will had a little complication to the gen2 subconstructors.

[raybellis]

I've been restarting Ghidra between each test anyway, as I didn't know about ReloadSleighLanguage :)

The binary (for an E-mu Emax sampler) is here

I suspect I've bitten off more than I can chew with the NS32016 as a first project for Sleigh... (but thanks for the support!)

[GhidorahRex]

It's definitely a do-able project, just start small. Get flow control andMOVB and all it's addressing modes first, then worry about the other instructions. It's not a huge instruction set.

What you'll need to do is add a context variable to accommodate the number of gen1 bytes that will be consumed. Then each size needs its own skipbyte constructor. Something like:

define context contextreg
  isize=(0,1) noflow
  skipb=(2,4) noflow
;

...

skipbytes: is skipb=0 & i16_all {}
skipbytes: is skipb=1 & i16_all; val8 {}
skipbytes: is skipb=2 & i16_all; val16 {}
skipbytes: is skipb=3 & i16_all; val32 {}

disp1:  val8 is  d8_mask=0b0  &  val8 [skipb=1;] { tmp:4 = val8; export tmp; }
disp1: val16 is d16_mask=0b10 & val16 [skipb=2;] { tmp:4 = val16; export tmp; }
disp1: val32 is d32_mask=0b11 & val32 [skipb=3;] { tmp:4 = val32; export tmp; }

disp2:  val8 is  d8_mask=0b0  &  val8 { tmp:4 = val8; export tmp; }
disp2: val16 is d16_mask=0b10 & val16 { tmp:4 = val16; export tmp; }
disp2: val32 is d32_mask=0b11 & val32 { tmp:4 = val32; export tmp; }

...

gen1: i16_gen1_r is i16_gen1_op=0b00 & i16_gen1_r [ skipb=0; ]{ export i16_gen1_r; }
gen1: @disp1 is (i16_gen1=0b10101); disp1 { build disp1; export disp1; }

...

gen2: @disp2 is ((i16_gen2=0b10101) ... & skipbytes); disp2 { build skipbytes; export disp2; }

:MOV^i  gen1,gen2 is (i & fmt4_op=0b0101) ... & gen1 ... & gen2 { build gen1; build gen2; }

With that I got:

Also you'll want separate MOVB, MOVW, MOVD constructors to get the correct memory sizing. Same with pretty much all of the other ^i constructors, but the isize stuff is fine for getting the correct immediate sizing (using the skipbytes stuff for that too). You'll want separate imm tables for source and dest for the skipbytes. For the PC + disp you'll want to use either inst_start or inst_next instead of PC (see most processors - they all have relative addressing using a reloc variable).