Multimodal AI Agents: Beyond Text Generation

What multimodal means here

A multimodal agent works with more than one kind of input inside a single task — text, images, audio, spreadsheets, slides, PDFs — rather than only generating text from text.

The practical significance is not the technical capability but what it removes. Business material does not arrive as clean text. It arrives as a scanned invoice, a recorded call, a spreadsheet with merged cells, a deck someone exported to PDF, a photograph of a whiteboard. A text-only system requires a human to convert all of that into text first, and that conversion is usually where the time actually goes.

So multimodality mostly buys the removal of retyping — which sounds mundane and is, in most organisations, the largest single share of the manual effort in knowledge work.

Where it matters in practice

• Documents that were never text. Scanned contracts, signed forms, invoices, delivery notes — read directly instead of transcribed.
• Charts and tables inside documents. A figure in a PDF report carries the finding; a text-only extraction loses exactly the part that mattered.
• Meetings. Audio to transcript to decisions, owners and follow-up.
• Spreadsheets. Structure and formulae, not just the visible values.
• Presentations. Both reading them as source material and producing them as output.

The common thread is that the format is incidental to the content. The value is in reaching the content without a person acting as a converter first.

From a meeting to a document

The chain that makes multimodality concrete: a recorded meeting becomes a transcript with speakers separated; the transcript becomes a set of decisions, owners and open questions; those become tasks and a written summary; the summary enters the knowledge base, where it can inform later work.

Almost all of the value sits after the transcript. A transcript on its own is a long document nobody reads. The decisions, the tasks and the searchable record are what change how the organisation works — and each of those steps is a place where an agent can be right or wrong, so each is a place worth reviewing.

Transcription quality is worth being honest about. It is good enough to work from and not good enough to publish unread: accuracy degrades with overlapping speech, strong accents, dialect and specialist vocabulary. Treat a transcript as a draft, and treat anything derived from it — especially attributed decisions — as needing a check by someone who was in the room.

What it still does badly

Multimodal capability is uneven, and the failure modes are specific rather than general.

Dense tables in scanned documents. Column alignment is frequently misread, and the error is silent — the numbers look plausible and are attached to the wrong row.

Handwriting. Usable for clear print, unreliable for cursive and annotations.

Charts without underlying data. A model reads a trend from a chart image; it does not read precise values, and asking it for them invites confident invention.

Audio with several speakers at once. Speaker attribution degrades exactly when a meeting gets interesting.

None of these argue against using multimodal agents. They argue for putting review where the failure is silent — which is a different placement from where review is usually put, and the reason knowing the limits is more useful than knowing the capabilities.