Skip to content

Unintended loop when using select with to parameters #2454

@aparcar

Description

@aparcar

Describe the bug

I'm filtering through a short YAML file and convert it to JSON, ever since I added another parameter to the select function some unexpected loop triggers and the output becomes 1000s of lines instead of the expected ~30. With just one parameter, things work fine.

Version of yq: 4.47.1
Operating system: mac
Installed via: homebrew

Input Yaml
I tried to reproduce it with a subset but it did not trigger the bug, so here the full document.

data1.yml:

devices:
  genexis_pulse-ex400:
    name: Genexis Pulse EX400 / Inteno Pulse EX400
    target: ramips-mt7621
    firmware: initramfs-kernel.bin

  hpe_msm460:
    name: Hewlett-Packard MSM460
    target: mpc85xx-p1020
    firmware: initramfs-kernel.bin

  openwrt_one:
    name: OpenWrt One
    target: mediatek-filogic
    firmware: initramfs.itb

  tplink_tl-wdr3600-v1:
    name: TP-Link TL-WDR3600 v1
    target: ath79-generic
    firmware: initramfs-kernel.bin

  tplink_tl-wdr4300-v1:
    name: TP-Link TL-WDR4300 v1
    target: ath79-generic
    firmware: initramfs-kernel.bin

  tplink_tl-wr842n-v3:
    name: TP-Link TL-WR842N v3
    target: ath79-generic
    firmware: initramfs-kernel.bin

  tplink_tl-wr1043nd-v3:
    name: TP-Link TL-WR1043N/ND v3
    target: ath79-generic
    firmware: initramfs-kernel.bin

  cznic_turris-omnia:
    name: CZ.NIC Turris Omnia
    target: mvebu-cortexa9
    firmware: initramfs-kernel.bin

  xunlong_orangepi-zero2:
    name: Orange Pi Zero2
    target: sunxi-cortexa53
    firmware: initramfs-kernel.bin
    snapshots_only: true

  bananapi_bpi-r4:
    name: Bananapi BPi-R4
    target: mediatek-filogic
    firmware: initramfs-recovery.itb

  bananapi_bpi-r64:
    name: Bananapi BPi-R64
    target: mediatek-mt7622
    firmware: initramfs-recovery.itb

  glinet_gl-mt1300:
    name: GL.iNet GL-MT1300
    target: mediatek-mt7621
    firmware: initramfs-kernel.bin

  glinet_gl-mt6000:
    name: GL.iNet GL-MT6000
    target: mediatek-filogic
    firmware: initramfs-kernel.bin

  rpi-4:
    name: Raspberry Pi 4B/400/CM4 (64bit)
    target: bcm27xx-bcm2711
    firmware: squashfs-factory.img.gz

  enterasys_ws-ap3710i:
    name: Enterasys WS-AP3710i
    target: mpc85xx-p1020
    firmware: initramfs-kernel.bin

  iei_puzzle-m902:
    name: iEi Puzzle-M902
    target: mvebu-cortexa72
    firmware: initramfs-kernel.bin

labs:
  labgrid-aparcar:
    proxy: labgrid-aparcar
    maintainers: "@aparcar"
    devices:
      - openwrt_one
      - tplink_tl-wdr3600-v1
      - genexis_pulse-ex400
      - bananapi_bpi-r4
      - glinet_gl-mt6000
      - enterasys_ws-ap3710i
    developers:
      - aparcar
      - dangowrt
      - jonasjelonek

  labgrid-bastian:
    proxy: labgrid-bastian
    maintainers: "@bittorf"
    devices:
      - cznic_turris-omnia
      - tplink_tl-wdr4300-v1
    developers:
      - aparcar

  labgrid-blocktrron:
    proxy: labgrid-blocktrron
    maintainers: "@blocktrron"
    devices:
      - hpe_msm460
      - tplink_tl-wr842n-v3
    developers:
      - aparcar

  labgrid-leinelab:
    proxy: labgrid-leinelab
    maintainers: "@lemoer"
    devices:
      - tplink_tl-wr1043nd-v3
    developers:
      - aparcar

  labgrid-hsn:
    proxy: labgrid-hsn
    maintainers: "@jonasjelonek"
    devices:
      - genexis_pulse-ex400
      - bananapi_bpi-r64
      - bananapi_bpi-r4
      - iei_puzzle-m902
    developers:
      - aparcar
      - jonasjelonek

  labgrid-wigyori:
    proxy: labgrid-wigyori
    maintainers: "@wigyori"
    devices:
      - xunlong_orangepi-zero2
    developers:
      - wigyori

developers:
  aparcar:
    sshkey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDZTxjQ3/KTefKvsPlVBWz+ITD7dGWcOt8/C55ekd2VE
  dangowrt:
    sshkey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDe6tUbVJW3bYUVCYyQQFpNrJ1wP5kzvEkCruSjyA6TM
  jonasjelonek:
    sshkey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDy4T0osxR8yS4RjJcsMn3rSacKvNzu+ZRNfe2lXTSHS
  wigyori:
    sshkey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG8vcHMKcl6W6OuzSdjOhQ0ZehtH+McjLB7WzgP9P4Ls

Command

The problematic part is $devices[.] != null and $devices[.].snapshots_only != true, with just $devices[.] != null it works fine.

The command you ran:

. as $root |
  $root.labs as $labs |
  $root.devices as $devices |
  $labs
  | to_entries
  | map(
      .key as $lab |
      .value.devices
      | map(
          select($devices[.] != null and $devices[.].snapshots_only != true) |
          {
            "device": .,
            "proxy": $labs[$lab].proxy,
            "target": $devices[.].target,
            "firmware": $devices[.].firmware,
            "maintainers": $labs[$lab].maintainers,
            "snapshots_only": ($devices[.].snapshots_only // false)
          }
        )
    )
  | flatten

Actual behavior

Runs for a long time, then spits out thousands of lines in the expected format, but nonsense repetitions.

Expected behavior

I convert things via -o=json to JSON

[
  {
    "device": "openwrt_one",
    "proxy": "labgrid-aparcar",
    "target": "mediatek-filogic",
    "firmware": "initramfs.itb",
    "maintainers": "@aparcar",
    "snapshots_only": false
  },
  {
    "device": "tplink_tl-wdr3600-v1",
    "proxy": "labgrid-aparcar",
    "target": "ath79-generic",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@aparcar",
    "snapshots_only": false
  },
  {
    "device": "genexis_pulse-ex400",
    "proxy": "labgrid-aparcar",
    "target": "ramips-mt7621",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@aparcar",
    "snapshots_only": false
  },
  {
    "device": "bananapi_bpi-r4",
    "proxy": "labgrid-aparcar",
    "target": "mediatek-filogic",
    "firmware": "initramfs-recovery.itb",
    "maintainers": "@aparcar",
    "snapshots_only": false
  },
  {
    "device": "glinet_gl-mt6000",
    "proxy": "labgrid-aparcar",
    "target": "mediatek-filogic",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@aparcar",
    "snapshots_only": false
  },
  {
    "device": "enterasys_ws-ap3710i",
    "proxy": "labgrid-aparcar",
    "target": "mpc85xx-p1020",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@aparcar",
    "snapshots_only": false
  },
  {
    "device": "cznic_turris-omnia",
    "proxy": "labgrid-bastian",
    "target": "mvebu-cortexa9",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@bittorf",
    "snapshots_only": false
  },
  {
    "device": "tplink_tl-wdr4300-v1",
    "proxy": "labgrid-bastian",
    "target": "ath79-generic",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@bittorf",
    "snapshots_only": false
  },
  {
    "device": "hpe_msm460",
    "proxy": "labgrid-blocktrron",
    "target": "mpc85xx-p1020",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@blocktrron",
    "snapshots_only": false
  },
  {
    "device": "tplink_tl-wr842n-v3",
    "proxy": "labgrid-blocktrron",
    "target": "ath79-generic",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@blocktrron",
    "snapshots_only": false
  },
  {
    "device": "tplink_tl-wr1043nd-v3",
    "proxy": "labgrid-leinelab",
    "target": "ath79-generic",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@lemoer",
    "snapshots_only": false
  },
  {
    "device": "genexis_pulse-ex400",
    "proxy": "labgrid-hsn",
    "target": "ramips-mt7621",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@jonasjelonek",
    "snapshots_only": false
  },
  {
    "device": "bananapi_bpi-r64",
    "proxy": "labgrid-hsn",
    "target": "mediatek-mt7622",
    "firmware": "initramfs-recovery.itb",
    "maintainers": "@jonasjelonek",
    "snapshots_only": false
  },
  {
    "device": "bananapi_bpi-r4",
    "proxy": "labgrid-hsn",
    "target": "mediatek-filogic",
    "firmware": "initramfs-recovery.itb",
    "maintainers": "@jonasjelonek",
    "snapshots_only": false
  },
  {
    "device": "iei_puzzle-m902",
    "proxy": "labgrid-hsn",
    "target": "mvebu-cortexa72",
    "firmware": "initramfs-kernel.bin",
    "maintainers": "@jonasjelonek",
    "snapshots_only": false
  }
]

Additional context

When I run the command in a pipe via yq -o=json labnet.yaml | jq <command>, things work just fine.

Sorry for the lengthy example

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions