fly.toml

[mkagit]

Could you please help to build a fly.toml using the Docker image. And how to run more than one bot in the same Docker.

[chilir]

The Docker image is meant to be a thin wrapper around a single instance of the application process, so you wouldn't be able to run multiple instances of the app within a single container.

I'm not too familiar with fly.io, but after looking at their docs and consulting with some LLMs, I think you could try deploying two bots as two separate processes with a single deployment. There are some prerequisite steps:

• Instead of passing in a .env file, you would need to pass in sensitive environmental variables (such as DISCORD_BOT_TOKEN, API_KEY, etc.) as secrets using their CLI tool (i.e. fly secrets)
• All mount volumes need to be created using the their CLI tool beforehand in the same region (i.e. fly volumes create <volume_name> --region <region>)
• Each bot's config.yaml file needs to be copied their respective config volume (i.e. using fly ssh sftp)

# fly.toml

app = "chat-thyme-two-bots"  # rename as needed
primary_region = "iad"  # change to your region

[build]
image = "ghcr.io/chilir/chat-thyme:v0.1.0"

[processes]
bot1 = "-c /app/config/config.yaml"
bot2 = "-c /app/config/config.yaml"

[[mounts]]
source = "bot1_config"  # volume needs to be created, bot1 config.yaml file also needs to be added to this volume beforehand
destination = "/app/config"
processes = ["bot1"]

[[mounts]]
source = "bot1_logs"  # volume needs to be created beforehand
destination = "/app/logs"
processes = ["bot1"]

[[mounts]]
source = "bot1_sqlite"  # volume needs to be created beforehand
destination = "/app/.sqlite"
processes = ["bot1"]

[[mounts]]
source = "bot2_config"  #  volume needs to be created, bot2 config.yaml file also needs to be added to this volume beforehand
destination = "/app/config"
processes = ["bot2"]

[[mounts]]
source = "bot2_logs"  # volume needs to be created beforehand
destination = "/app/logs"
processes = ["bot2"]

[[mounts]]
source = "bot2_sqlite"  # volume needs to be created beforehand
destination = "/app/.sqlite"
processes = ["bot2"]

Like I mentioned, I've never used fly.io and I'm not familiar with their tooling, apologies in advance if this deployment configuration doesn't work out of the box. I'm not sure if I would be much help in debugging fly.io deployment issues.

Another potential way to go about it is to split the deployment into two separate deployments (one for each bot) and directly pass in chat-thyme configuration as environmental variables (sensitive ones from .env should still be added as secrets) OR command line args. This isn't well documented yet, but every configuration option in config.yaml can actually be set through the command line and environmental variables as well, refer to this snippet within the codebase:

chat-thyme/src/config/parse.ts

Lines 156 to 212 in a286a30

	// Priority:
	// 1. Command line arguments
	// 2. Environment variables
	// 3. .env
	// 4. YAML config
	// 5. Default values
	const rawConfig = {
	discordBotToken: process.env.DISCORD_BOT_TOKEN,
	model:
	parsedArgs.model ??
	(process.env.CHAT_THYME_MODEL !== undefined
	? process.env.CHAT_THYME_MODEL
	: parsedFromConfig.model),
	serverUrl:
	parsedArgs.serverUrl ??
	(process.env.MODEL_SERVER_URL !== undefined
	? process.env.MODEL_SERVER_URL
	: parsedFromConfig.serverUrl),
	apiKey: process.env.API_KEY ?? parsedFromConfig.apiKey,
	useTools:
	parsedArgs.useTools ??
	(process.env.USE_TOOLS !== undefined
	? process.env.USE_TOOLS
	: parsedFromConfig.useTools),
	exaApiKey: process.env.EXA_API_KEY ?? parsedFromConfig.exaApiKey,
	systemPrompt:
	parsedArgs.systemPrompt ??
	(process.env.MODEL_SYSTEM_PROMPT !== undefined
	? process.env.MODEL_SYSTEM_PROMPT
	: parsedFromConfig.systemPrompt),
	dbDir:
	parsedArgs.dbDir ??
	(process.env.DB_DIR !== undefined
	? process.env.DB_DIR
	: parsedFromConfig.dbDir),
	dbConnectionCacheSize:
	parsedArgs.dbConnectionCacheSize ??
	(process.env.DESIRED_MAX_DB_CONNECTION_CACHE_SIZE !== undefined
	? process.env.DESIRED_MAX_DB_CONNECTION_CACHE_SIZE
	: parsedFromConfig.dbConnectionCacheSize),
	dbConnectionCacheTtl:
	parsedArgs.dbConnectionCacheTtl ??
	(process.env.DB_CONNECTION_CACHE_TTL_MILLISECONDS !== undefined
	? process.env.DB_CONNECTION_CACHE_TTL_MILLISECONDS
	: parsedFromConfig.dbConnectionCacheTtl),
	dbConnectionCacheEvictionInterval:
	parsedArgs.dbConnectionCacheEvictionInterval ??
	(process.env.DB_CONNECTION_CACHE_EVICTION_INTERVAL_MILLISECONDS !==
	undefined
	? process.env.DB_CONNECTION_CACHE_EVICTION_INTERVAL_MILLISECONDS
	: parsedFromConfig.dbConnectionCacheEvictionInterval),
	discordSlowModeInterval:
	parsedArgs.discordSlowModeInterval ??
	(process.env.DISCORD_SLOW_MODE_SECONDS !== undefined
	? process.env.DISCORD_SLOW_MODE_SECONDS
	: parsedFromConfig.discordSlowModeInterval),
	};

[mkagit]

Thank you very much for trying to set up Fly for me. I think Fly is an excellent platform for running bots. However, since I had difficulty deploying it, I first set it up on my local Docker Desktop.
At first, I struggled a bit to understand the concept—how you prepared the bot to work and what the overall idea is. But I think I now understand that you want it to be a bot focused on temporary, topic-based conversations with search capabilities.
I created an EXA key for myself, and the bot recognized it. However, on my first attempt, it executed the search, but when returning the results, it gave a 400 error. I am testing with models from Groq and OpenRouter as well.
For now, I think you could make the concept and intended functionality of the bot clearer in the repository. I understood that it works differently from most bots out there. Most bots function like a user you call upon to ask anything throughout a conversation, whereas yours is more focused on private discussions with tool usage.
I also noticed that when starting a conversation, there are many options appearing in the menu—it seems like there are three options. I think that’s a bit too much to look at right away. It would be better to have this pre-configured in the config file.
Another thing I believe people will ask about is how to use multiple models and have the option to choose between them when chatting.
Lastly, I received very long responses, which were split into three or four messages, and I think that’s not very comfortable in Discord. It would be better if the responses were limited to a single message instead.

[chilir]

Thanks for the feedback, appreciate it.

but when returning the results, it gave a 400 error. I am testing with models from Groq and OpenRouter as well.

Could you let me know which models were returning the errors? Maybe I could set up an issues template for others to report similar issues on a case-by-case basis. Also it's possible that some unsupported parameters were passed in to the chat completions request which could also cause 400 errors, you could try using /start-chat strict True to filter out parameters that are not part of the OpenAI chat completion API.

Based on my own testing, there are some caveats with tool use:

• Reasoning models across the board generally won't work with tool use, or at least not multi-turn tool use through the OpenAI chat completions API
• (non-reasoning) Llama3-based models and OpenAI models should generally work out of the box with multi-turn tool use
• Gemini models will not work at all with multi-turn tool use through the OpenAI chat completions API

Since I had limited testing, I had hoped Gemini was the exception to the norm. But if it turns out that most provider-hosted models across the board are unable to support multi-turn tool use through the OpenAI chat completions API, I may need to make multi-turn tool use a configurable option, document which model families are known to work with it, and default to some sort of scaffolding around single turn tool calls. Gemini for example is able to make the tool call, but it's unable to process the tool call response so it just throws out 400 errors.

I also noticed that when starting a conversation, there are many options appearing in the menu—it seems like there are three options. I think that’s a bit too much to look at right away. It would be better to have this pre-configured in the config file.

That's fair, originally my thought was that users may want to have easy access to configure sampling parameters (e.g. top P, frequency penalty, etc.) on a per-chat basis. Those slash command options are optional and if not specified, they won't be included in the chat completion request to the model (so it'll default to whatever is set up in the model). But I can definitely see how it's a bit confusing and overwhelming. What I could do is set up separate slash commands for each tunable sample parameter and move them out of the options for the /start-chat and /resume-chat commands. I'll put this on the backlog 🚧.

Lastly, I received very long responses, which were split into three or four messages, and I think that’s not very comfortable in Discord. It would be better if the responses were limited to a single message instead.

Unfortunately Discord messages have a hard character limit of 2000 characters. I suppose the way around this is to put model responses in a paginated embed instead of a message. I'll also throw this on the backlog 🚧.