Skip to content

feat: add token-based output splitting#21

Merged
dyoshikawa merged 1 commit intodyoshikawa:mainfrom
dyoshikawa-claw:feat/max-tokens-output-splitting
Mar 15, 2026
Merged

feat: add token-based output splitting#21
dyoshikawa merged 1 commit intodyoshikawa:mainfrom
dyoshikawa-claw:feat/max-tokens-output-splitting

Conversation

@dyoshikawa-claw
Copy link
Copy Markdown
Contributor

Summary

  • add --max-tokens to split rendered JSON output by token count
  • keep output chunking in date order while honoring the stricter of size/token limits
  • document the new behavior and add tests for token-based chunking

Details

  • token counting uses github.com/tiktoken-go/tokenizer with o200k_base
  • if a single rendered event exceeds --max-tokens, the command returns an error

Validation

  • go mod tidy
  • go test ./...
  • go build ./cmd/ghactivities

@dyoshikawa dyoshikawa merged commit f45546b into dyoshikawa:main Mar 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants