Building an expert agent with Hugging Face, in sixteen short pieces

The short version: An agent is a language model in a loop, with tools it can call and the ability to decide what to do next. That last part — the model choosing the steps — is the whole distinction.

The word "agent" gets applied to everything, so here's a spectrum that actually separates things:

An agent has four parts, and everything in this file is about one of them:

• A model — the reasoning engine (sections 5, 6, 12)
• Instructions — who it is, what it does, what it refuses (sections 3, 7)
• Tools — the actions it can take in the world (sections 10, 11)
• Knowledge — the domain material it can consult (sections 8, 9)

The short version: Hugging Face isn't one product. It's a hub of models and datasets plus a stack of libraries and hosting options, and knowing which piece does what saves a lot of wandering.

The short version: "Expert agent" means it does a narrow thing very well and knows what it isn't. Write down the boundary before you write any code — most failed agent projects skipped this.

Answer these four before touching a library. They determine every later decision, and none of them is a technical question:

• What questions is it for? Write ten real ones, in the words a real user would use — not idealized versions. These become your evaluation set in section 13, so write them down properly.
• What does it have access to that a general model doesn't? Your documents? A live database? A calculation? If the answer is "nothing," you don't need an agent — you need a good prompt on a general model, and that's a legitimate and much cheaper outcome.
• What should it refuse or escalate? An expert agent that answers everything isn't an expert; it's a chatbot with a job title. Being able to say "that's outside what I cover, here's who to ask" is a feature you have to design in, because models default to attempting everything.
• What does being wrong cost? A recipe suggester and a compliance assistant need completely different levels of verification, citation, and human review. Decide this now, because it sets how much of sections 13 and 15 you actually need.

The short version: Prompting, retrieval, and fine-tuning. Nearly everyone reaches for fine-tuning first and should reach for it last.

The practical sequence: prompt until it stops improving, add retrieval if it needs facts it doesn't have, and only consider fine-tuning when you can point to a specific behavior that prompting can't produce. Most good expert agents are prompting plus retrieval, full stop.

The short version: Pick an instruction-tuned model, sized to your hardware and budget, with a license that permits your use. Then test two or three on your own questions, because leaderboards won't settle it.

The Hub has a great many models. Filters that narrow it fast:

• Instruct, not base. Model names ending in -Instruct or -Chat have been trained to follow instructions. A base model completes text and will not behave as you expect in a conversation. This trips up nearly every beginner once.
• Size vs. where it runs. A rough guide: parameters in billions × 2 = GB of memory at 16-bit, halved again at 8-bit and again at 4-bit (quantization). A 7–8B model quantized to 4-bit fits comfortably on consumer hardware. A 70B model does not, without serious equipment.
• License. Check it before you build. Some open-weight models carry commercial restrictions or acceptable-use terms; some are gated and require accepting terms on the model page. Discovering this after you've built is an expensive way to learn.
• Agent capability specifically. Not all capable models are good at tool use. If the agent must reliably emit well-formed calls, that's a distinct skill — look for models that advertise function-calling or tool-use training, and test it early.
• Context window. How much retrieved material plus conversation plus instructions can fit. Matters a lot for RAG.

The short version: Three options — serverless API, dedicated endpoint, or your own machine. They trade cost, control, and privacy against each other, and you can switch later if you keep them behind one interface.

The serverless version is about this simple:

from huggingface_hub import InferenceClient

client = InferenceClient()          # uses your saved HF token
out = client.chat.completions.create(
    model="<org>/<model-name>",
    messages=[{"role": "user", "content": "Hello"}],
)
print(out.choices[0].message)

The short version: The system prompt is your agent's job description. Specific beats long, examples beat adjectives, and the boundaries matter more than the capabilities.

A working structure, in roughly this order:

1. Role and scope. What it is, what domain, who it serves.
2. Procedure. The steps it should follow — especially "search the knowledge base before answering," which models will skip if not told.
3. Output format. Exactly what a good answer looks like. Show one.
4. Boundaries. What's out of scope and what to do instead — the escalation path from section 3.
5. Uncertainty rule. What to say when the retrieved material doesn't cover the question. This line does more work than any other.

Three habits that reliably improve results

• Show, don't describe. One or two complete examples of an ideal exchange will beat three paragraphs describing the ideal. This is the highest-leverage edit available.
• Put the refusal case in an example too. Show it declining an out-of-scope question gracefully. Otherwise it will only have seen examples of answering, and will answer.
• Version it like code. Keep the system prompt in a file, in version control, with a note about what each change was meant to fix. You'll make dozens of revisions and you will otherwise lose track of which one helped.

The short version: Retrieval means splitting your documents into pieces, converting each into a vector that captures its meaning, and searching by meaning rather than by keyword. How you split matters more than which embedding model you use.

The pipeline has four steps, and the first is where most quality is won or lost:

1. Load and clean. Get documents into plain text. Strip navigation, boilerplate, and headers. Garbage in this stage is garbage forever.
2. Chunk. Split into retrievable pieces, ideally along natural boundaries — sections, headings, paragraphs — rather than every N characters. Keep some overlap so a sentence spanning a boundary isn't orphaned. And attach metadata to every chunk: source document, section, date. You need it for citations in section 13.
3. Embed. Run each chunk through an embedding model to get a vector. Browse the Hub's embedding leaderboards; a small, fast model is usually the right starting point.
4. Index. Store the vectors somewhere searchable. A local vector index is entirely adequate for tens of thousands of chunks — you don't need a database service to start.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("<embedding-model-id>")
vectors = model.encode(chunks, normalize_embeddings=True)
# then store vectors + chunk text + metadata in your index

The short version: When a RAG agent gives a bad answer, retrieval is the culprit far more often than the model. Test retrieval on its own, before blaming anything else.

The generation step can only work with what it was handed. If the right chunk wasn't retrieved, no model and no prompt will save the answer — it will simply produce a fluent answer from the wrong material, which is worse than an error because it looks fine.

So evaluate the two halves separately. Take twenty real questions, note which chunk should be retrieved for each, and measure how often it appears in the top results. That number is the ceiling on your agent's accuracy, and it's the number to improve first.

Fixes, roughly in order of payoff

• Hybrid search. Combine semantic (vector) search with old-fashioned keyword search. Vectors are bad at exact identifiers — part numbers, section references, rare proper nouns — and keyword search is excellent at precisely those. Together they cover each other's failures.
• Reranking. Retrieve 30 candidates cheaply, then run a cross-encoder reranker over them to pick the best 5. This is usually the single largest quality gain available, for modest added latency.
• Query rewriting. Have the model turn a conversational question into a better search query before retrieving. "What about returns?" is a poor query; "return window policy for online orders" is a good one.
• Tune top-k. Too few chunks and you miss the answer; too many and the relevant one gets buried in noise. Between three and ten is the usual range — test it rather than guessing.

The short version: A tool is a function the model can call. The function body is ordinary code; the part that determines whether it works is the description, because that's all the model sees.

In smolagents, a decorator and a docstring are the whole interface:

from smolagents import tool

@tool
def check_inventory(sku: str) -> str:
    """Look up current stock for a product SKU.

    Args:
        sku: The product SKU, e.g. "AB-1234". Case-insensitive.
    """
    return warehouse_api.get_stock(sku)

That docstring and those type hints get shown to the model. The model has never seen your code, your database, or your intentions — it decides whether to call this and what to pass based entirely on those few lines.

Principles that hold up

• Few and well-named beats many and overlapping. Two tools that both sort of search will get confused. Give each a distinct job and say in the description when not to use it.
• Return text a model can read. Not a raw JSON dump of 400 fields. Return the five fields that matter, formatted plainly. What you return goes straight into the context window and costs tokens and attention.
• Handle errors as messages, not exceptions. Returning "No product found with SKU AB-1234; check the format" lets the agent recover on the next step. A crash ends the run.
• Make destructive tools require confirmation, or don't give them at all. Section 15.

The short version: Think, act, observe, repeat. smolagents offers two flavors — one where the model writes Python to act, one where it emits structured tool calls — and the choice affects how you sandbox it.

Every agent framework implements the same cycle: the model reasons about the goal, chooses an action, the framework executes it, the result comes back into context, and round it goes until the model produces a final answer or hits the step limit.

Assembling it is genuinely short:

from smolagents import CodeAgent, InferenceClientModel

agent = CodeAgent(
    tools=[check_inventory, search_knowledge_base],
    model=InferenceClientModel(model_id="<org>/<model>"),
    max_steps=6,
)
agent.run("Is the AB-1234 still under the extended warranty terms?")

The short version: Later than you think, and for behavior rather than facts. When you do, use parameter-efficient methods — you almost never need to retrain a whole model.

From section 4: fine-tuning teaches how to respond, not what is true. Good reasons to do it:

• You need a consistent, unusual output format that prompting keeps drifting from
• Your domain has idiom and conventions that general models get subtly wrong
• You want a small model to do one narrow task as well as a large one — which is often the real payoff, because it cuts inference cost dramatically
• Your prompt has grown so long with instructions and examples that it's expensive on every single call

Bad reasons: "it doesn't know about our products" (retrieval), "it's not smart enough" (bigger model), "the facts are out of date" (retrieval, and they'll be out of date again next month).

What it actually takes

• Data. Hundreds to low thousands of high-quality examples in the exact input-output shape you want. Quality dominates quantity, and assembling this honestly is 80% of the work — the training itself is often an afternoon.
• Tools. The peft library for the adapters, trl for the training loop, datasets for the data. There are also no-code options if you'd rather not write the loop.
• A held-out test set you never train on, plus your section 13 evaluations. Without these you cannot tell whether fine-tuning helped, and "it feels better" is not a measurement.

The short version: Without an evaluation set you aren't improving your agent — you're changing it and hoping. Build the set before you build the agent, and make it boring and repeatable.

This is the section people skip and the one that separates a demo from something usable. The minimum viable version is a spreadsheet:

Thirty to fifty rows is plenty to start, and crucially, include out-of-scope and adversarial rows — questions it should decline, questions with false premises, questions phrased to get it to ignore its instructions. An agent evaluated only on questions it should answer will pass its test and fail in the world.

Then measure the three layers separately, because they fail for different reasons:

• Retrieval — was the right chunk in the top results? (Section 9.)
• Tool use — did it call the right tool with sensible arguments?
• Final answer — correct, properly cited, right format, appropriately refused when it should be?

The short version: A Space with a Gradio interface is the fastest path from working script to shareable URL, and it's free at the lowest tier.

The shortest route: write your agent in app.py, wrap it in a Gradio interface, add a requirements.txt, push to a new Space. It builds and serves itself at a public URL. For a CPU-only agent calling a hosted model, the free tier is genuinely sufficient.

import gradio as gr

def respond(message, history):
    return agent.run(message)

gr.ChatInterface(respond).launch()

The details that matter once it's real

• Secrets go in Space settings, never in the code. API keys committed to a public repo get found and abused within hours — this is not hypothetical.
• Public vs private. A public Space is visible to everyone and anyone can run it, which means anyone can spend your inference budget. If your agent touches private data or a paid API, make it private or add authentication.
• Spaces sleep. Free Spaces pause after inactivity and take time to wake, which is fine for a demo and bad for something people depend on. Persistent uptime is a paid tier.
• Storage isn't persistent by default. Files written by a running Space can vanish on restart. Anything that must survive — a vector index, logs, uploaded documents — belongs in a Dataset repo or a persistent storage option, not the local filesystem.
• Rebuild the index elsewhere. Build your embeddings offline, push them to a Dataset repo, and have the Space load them at startup. Re-embedding on every boot is slow and wasteful.

The short version: An agent reads untrusted text and takes actions based on it. That combination is the security problem, and it doesn't have a clean solution — it has controls.

Then the specific controls:

• Sandbox generated code. If you use a code-writing agent, run its code in an isolated sandbox — a container, or one of the remote sandbox services the framework supports — rather than on the machine that holds your files and tokens. The local interpreter is convenient for development and not a deployment posture.
• Scope every credential narrowly. Read-only tokens for read-only tools. A separate key for the agent so you can revoke it without touching anything else.
• Log everything. Every run, every tool call, every retrieved chunk. When someone reports a wrong answer next month, this is the only way to find out what happened.
• Mind the data leaving. If you're calling a hosted provider, your users' text goes to a third party. Check whether that's acceptable for your data — and if it isn't, that's the argument for a dedicated endpoint or local inference.

The short version: Build the dumbest version that touches every piece, get it end-to-end, then improve one layer at a time against your evaluation set.

The order matters because each step tells you whether the next one is needed:

1. Write the ten questions and the scope statement (section 3). Before any code.
2. Ask a strong hosted model those questions with no agent at all. If the answers are good, you're done — some tasks genuinely don't need this machinery, and finding that out costs twenty minutes.
3. Add the system prompt (section 7) and re-run. Note what improved.
4. Build the smallest possible knowledge base — ten documents, crude chunking — and check retrieval on its own (sections 8, 9). Not the answers yet, just whether the right chunks come back.
5. Wire up one tool and the agent loop (sections 10, 11). One tool. Watch the intermediate steps.
6. Turn your ten questions into a proper evaluation set and get a baseline score (section 13).
7. Now improve one thing at a time — reranking, better chunking, a different model, more examples in the prompt — measuring after each. One change per measurement, or you learn nothing.
8. Deploy to a private Space (section 14), use it yourself for a week, and fix what actually annoys you rather than what you imagined would.
9. Consider fine-tuning (section 12) only if a specific behavior is still wrong after all that.

Concrete enough to actually do, small enough to finish: an expert agent over a set of documents you know well. Your own writing, a manual, a body of regulations, a hobby's reference material — anything where you can tell instantly whether an answer is right, which is what makes it a good first project.

1. Collect 20–50 documents as plain text.
2. Write 15 questions you know the answers to, including three the documents don't cover. Those three are your refusal tests.
3. Chunk by section, embed with a small sentence-transformers model, index locally.
4. Write a search tool over that index and a system prompt that says: always search before answering, always cite the section, say plainly when the documents don't cover it.
5. Wire it into an agent with a hosted model, max_steps=5.
6. Run all 15 questions. Note which fail and why — retrieval or generation. That diagnosis is the skill.
7. Fix the biggest failure category. Re-run. Repeat twice.
8. Wrap in a Gradio chat interface and push to a private Space.

You'll have touched every concept in this file, and you'll have a real intuition for where the difficulty actually lives — which is almost never where beginners expect.

Where to learn more

Hugging Face's own free courses are the best starting point and are kept current: there's an agents course, an LLM course, and a deep-learning-for-NLP course, all on the Learn section of the site. The smolagents documentation includes guided tours and worked examples. And the most useful habit of all is browsing Spaces for projects near yours — most are open source, and reading someone's working app.py teaches more than any tutorial.

Since this area moves quickly, prefer official docs over blog posts, check the date on anything you read, and expect that any code older than about a year will need small fixes.