Support prompt caching and conversation memory for Anthropic API mode
What would you like?
I use EFKey's AI mode with my own Anthropic API key, and I'd love two options:
1. Prompt caching: let me add cache_control to the system prompt, so repeated calls with the same prompt cost less.
2. Conversation memory: let me choose to send the last few messages with each request, so rewrites stay consistent with my style.
A field to edit the raw request body would cover both.
Why it would help
I pay per token, and I send the same custom prompt on every rewrite. Caching would lower the cost of each call, and memory would help the model learn how I write over a conversation. Thanks for building a great keyboard.
Discussion
Sign in to comment or vote.
Caching wouldn't save you anything here, unfortunately. The whole system prompt EFKey sends is around 400 tokens even with a full custom prompt, and Anthropic only caches prompts from 4096 tokens on Haiku 4.5 (512 on the newest models). Below that, cache_control is silently ignored. Conversation memory would make every request more expensive, since the old texts get sent again each time, and the keyboard would have to store what you typed, which it deliberately never does. A raw request body field is something I won't add: one broken bracket and rewrite and translation stop working. If you want rewrites in your style, put a couple of lines you've written yourself into the custom action's prompt, like "write like this: …". The model sees them on every call. 2.2.2, the next App Store update, raises the custom prompt limit from 1,000 to 3,000 characters, so there's room for real samples.
Thanks for the clear explanation, that makes sense. Keeping typed text off the device is the right call, and I'd rather have reliable rewrites than a raw body field. I'll add a few of my own lines as style samples once 2.2.2 lands, and the 3,000 character limit gives me room for them. Happy to test it and report back.
Thanks!