Before comparing models, write down what you need from one. “Good at coding” is broad. “Explains an unfamiliar function accurately and proposes a change that passes these tests” is something you can check.
Make a tiny task set
Collect a few examples of real work: a document you can summarize yourself, a bug you understand, or a draft with clear editing requirements. Keep at least one example that often causes trouble. Do not include sensitive material in a service unless its settings and your permissions support that use.
Keep the comparison fair
Give each setup equivalent instructions and access to the same source material. Record which model and product you used. If one system can browse or execute code and the other cannot, you are comparing complete systems. That may be exactly the comparison you need, but label it accurately.
Score useful outcomes
- Correctness: Does it solve the actual problem?
- Evidence: Can you verify its facts, calculations, or changes?
- Effort: How much correction do you have to do?
- Time and cost: What does a completed task take, including retries?
These are suggested evaluation criteria, not a published benchmark. Repeat borderline examples rather than treating one impressive answer as proof.
Hosted or local?
For a local setup, add hardware capacity and response time to the comparison. The local model finder estimates memory fit; you still need to try your task. For a hosted product, check the actual model access, integrations, and service conditions you will use. Read the current provider information rather than relying on a family name.
Keep a reason to change
Save the tasks and the outcomes. When a new model arrives, rerun the cases that matter. Switching becomes a decision based on better work, not just a new name.