Models and usage

\n**A model picker and a meter**: ```razor @implements IAsyncDisposable ``` ```csharp private readonly BitAiUiOptions options = BitAiUiOptions.Default with { ShowUsage = true, ShowModelName = true, ShowTimestamps = true, }; private BitAiChatSession? session; private BitAiModel? model; private string? modelId; private readonly string[] suggestions = ["What did this turn cost?"]; // The catalog registered with ai.AddModels(...). [Inject] private IBitAiModelCatalog Catalog { get; set; } = default!; protected override void OnInitialized() { model = Catalog.DefaultModel; modelId = model?.ModelId; Rebuild(); } private async Task OnModelSelected(BitAiModel selected) { model = selected; var previous = session; Rebuild(); // The conversation carries over to the new session; the old one only has to stop. if (previous is not null) await previous.DisposeAsync(); } private void Rebuild() { if (model is null) return; var conversation = session?.Conversation ?? new BitAiConversation(); session = new BitAiChatSession( AiDemoInteractiveTransports.Metered(model), conversation, new BitAiChatSessionOptions { RequestOptions = new BitAiRequestOptions { ModelId = model.ModelId, Temperature = 0.7 }, }); } public ValueTask DisposeAsync() => session?.DisposeAsync() ?? ValueTask.CompletedTask; ``` `Program.cs`: ```csharp builder.Services.AddBitBlazorUIAIServices(ai => { ai.UseTransport(AiDemoTransports.Echo()); // The models the selector offers, the providers they are grouped under, and the default. ai.AddModels(AiDemoCatalog.Models, AiDemoCatalog.Providers, defaultModelId: "demo-model-1"); }); // AiDemoCatalog.Models holds three; this is one of them. The rates are what the cost under each answer is priced with. var demoLarge = new BitAiModel("demo-model-1", "Demo Large") { ProviderId = "demo", Family = "Demo Provider", Description = "The general model. Slower, better at long reasoning.", Capabilities = BitAiModelCapabilities.Streaming | BitAiModelCapabilities.Tools | BitAiModelCapabilities.Vision | BitAiModelCapabilities.Reasoning | BitAiModelCapabilities.StructuredOutput | BitAiModelCapabilities.Citations, ContextWindow = 200_000, MaxOutputTokens = 16_000, InputPricePerMillion = 3m, OutputPricePerMillion = 15m, }; // The providers the models are grouped under. var demoProvider = new BitAiProvider("demo", "Demo Provider") { Website = "https://example.com" }; ``` `AiDemoInteractiveTransports.cs`: ```csharp // AiScriptedTransport and AiScript are the demo's helpers - see the Basic chat scenario. public static IBitAiTransport Metered(BitAiModel model) => new AiScriptedTransport((request, token) => AiScript.Stream(token, async yielder => { await yielder(new BitAiMessageStartedEvent()); await yielder(new BitAiMetadataEvent(new BitAiGenerationMetadata { ModelId = model.ModelId, ProviderId = model.ProviderId, ContextWindow = model.ContextWindow, Temperature = 0.7, })); var answer = $"This turn ran on {model.Name}."; await foreach (var evt in AiScript.Write(answer, 4, 10, token)) await yielder(evt); var input = request.Messages.Sum(m => m.GetText().Length) / 4; var output = answer.Length / 4; // What the provider reports; the components work out the rest. await yielder(new BitAiUsageUpdatedEvent(new BitAiTokenUsage { InputTokens = input, OutputTokens = output, CachedInputTokens = input / 3, Cost = model.EstimateCost(new BitAiTokenUsage { InputTokens = input, OutputTokens = output }), Currency = model.Currency, })); await yielder(new BitAiMessageCompletedEvent()); })); ```

The models come from the catalog registered with ai.AddModels(models, providers, defaultModelId), which the selector reads (IBitAiModelCatalog), grouped by family or provider. Picking another one moves the conversation onto a new session for that model; the old session only has to stop.

The choice reaches the transport in the session's RequestOptions, a BitAiRequestOptions sent with every request (BitAiRequest.Options): here its ModelId and Temperature. It also carries ProviderId, TopP, MaxOutputTokens, StopSequences, Seed, SystemPrompt, the ToolChoice (Tool calling), the ResponseFormat (Structured output) and an Extra dictionary for provider settings with no field of their own. Every one is optional, and a transport ignores what its provider cannot express. A session's Options can also be replaced between turns, when the transport itself stays the same.

The transport reports back what it used: a BitAiMetadataEvent with the model, the provider, the context window and the temperature, merged into the reply's metadata, and a BitAiUsageUpdatedEvent with the input, output and cached tokens and the cost. Usage is cumulative within one stream - a later report replaces an earlier one, so a provider's running totals can be forwarded as they come - and the streams of one turn add up, one per tool round trip.

Every field of the accounting is optional, because providers report different subsets of it. The components do the arithmetic they can: throughput from the output tokens over the streaming window, cost from the rates in the model catalog (BitAiModel.EstimateCost), and the share of the context window the conversation uses from the window the transport or the model reports. ShowUsage, ShowModelName and ShowTimestamps of BitAiUiOptions decide what a message shows.