token-trim
A Python CLI tool that removes docstrings, comments, and redundant whitespace from source code before AI model processing.
Open productWhat it does
About token-trim
token-trim is a Python command-line utility designed to shrink code token counts before sending source code to AI models. Using Abstract Syntax Tree parsing and regeneration with ast.unparse(), token-trim safely strips comments, docstrings, and dead whitespace without syntax corruption. It counts token savings using tiktoken and supports directory walking as well as line-based cleanups for non-Python files.
- Availability:
- launched
- Access:
- unknown
- Format:
- software
- Platform:
- cli
Core capabilities
AST Code Pruning
Parses Python source files into syntax trees to safely strip docstrings, comments, and unnecessary whitespace while ensuring valid code outputs.
Tiktoken Savings Analysis
Measures precise token count reductions before and after code trimming using the tiktoken tokenizer.
Directory and Batch Trimming
Keep discovering
Related debuts
Continue with products that solve a similar problem or serve the same kind of maker.
