Estimated reading time at 200 wpm: 8 minutes
Well, for those few people with intelligence, they would have seen the work done in AI Model Asynchronous Batching – The Captain’s Watch on 27 October (yesterday). No one – certainly not me – expects everything to work well after a first round of development. This morning overnight batches with PDF attachments were stuck in the system, and no results of analysis would have ever returned. So a massive troubleshooting and development round followed, with me forcing collaboration between Gemini Argon and Claude Opus 5.5.
Whether or not you agree our Fat Disclaimer applies
Some may be wondering if this is re-inventing the wheel. Nope – this was actually inventing the wheel. How? Nobody – nobody on the internet has achieved this solution that works!
But the scene was like riding with two stallions pulling a coach – both very powerful – each thinking they have the right direction. Reigning them in was akin to a scene from the wild west – where a coach master had maintain course. The coach master isn’t stronger than either horse. He can’t out-pull them, and he never tries. He holds the reins and sees the whole road, while each horse only sees the pull in its own harness. Gemini Argon and Opus each had the strength to run me off a cliff, each fully convinced it was right — and each was right, within its own frame. The thing neither had was the other’s view, and neither had the whole route. I was the only one in arrangement holding both reins and the map.
Power doesn’t steer; judgment does. The stallions pull, the coach master decides when to let them run and when the correct answer to their argument is neither of you.
Details of the work follow – my work – I am the coach master. No AI model was going to get to success without my hard work!
Executive Summary
The live verification of OpenRouter Batch Dispatcher (dev 1.7.1) marks the successful transformation of an experimental asynchronous script into a robust, production-grade OpenWebUI pipeline. By integrating resilient multi-format document extraction (PDF, DOCX, TXT, and MD), thread-safe atomic ledger persistence, SHA-256 request fingerprinting, and rigorous error isolation, the architecture eliminates historical vulnerabilities concerning UI state duplication, fragile file resolution, and unhandled container exceptions. Live testing so far, confirms full control-plane stability, accurate multi-job parsing, and reliable background polling fidelity.
Part 1: The Initial Baseline
The work began with a straightforward operational objective: establish reliable asynchronous communication with OpenRouter’s batch endpoint from within an OpenWebUI environment. The immediate priority was translating the standard /v1/chat/completions request structure into a custom pipe capable of packaging bulk jobs and dispatching them under the provider’s twenty-four-hour processing window.
That first build handled basic text input adequately. It read user prompts, split execution blocks using simple model directives, and fired off POST requests with a basic timeout configuration. Polling mechanisms were bolted on to query OpenRouter using stored batch identifiers, retrieving raw completion arrays once the provider marked the queue complete.
Beneath that surface functionality, however, significant structural vulnerabilities remained exposed. File resolution relied on guessing internal container directory paths, creating a brittle dependency that would shatter during routine updates. The architecture possessed no mechanism to prevent duplicate dispatches during chat regenerations or message edits, leaving the pipeline open to redundant API costs from accidental double-clicks. Document handling lacked any capability to screen for scanned, text-free PDF files, and text extraction ignored table structures or footnotes in Word documents entirely. When errors occurred, the code risked spilling unhandled tracebacks directly into the user interface rather than isolating exceptions gracefully.
Part 2: Stress-Testing, Diagnostics & Collaborative Refinement
Morning auditing and real-world stress-testing exposed the operational vulnerabilities of the initial build, prompting a targeted collaborative overhaul to elevate the pipeline to production grade (dev_ver_1.7.1). The redesign addressed structural weaknesses across both the control and data planes through rigorous architectural additions:
- Control Plane Security & Ledger Integrity: To eliminate the risk of accidental double-dispatches during UI chat regenerations or message edits, a SHA-256 request fingerprinting mechanism was introduced. This maps the model and complete request payload against a local JSON ledger within a defined time window. Concurrency safety was secured using a thread lock coupled with atomic OS-level file replacement (
os.replace), ensuring that background polling updates or ledger writes never leave a half-written file state on disk. - Data Plane Hardening & File Ingestion: Reliance on speculative path-rebuilding was eliminated by mapping file extractions directly to native OpenWebUI dictionary objects supplied in the runtime environment. For PDFs, a proportional per-page density verification (< 100 characters per page) was implemented to automatically intercept and refuse scanned, text-free image documents alongside password-protected files. Word documents (
.docx) received advanced XML parsing to preserve document body order and table layouts while collapsing redundant text from merged cells, accompanied by automated extraction ofword/footnotes.xmlappended cleanly as a separate section. - Error Isolation & Operational Feedback: User-facing operational faults—such as missing files, unsupported formats, or empty job definitions—were isolated into a dedicated
InputErrorclass, surfacing as clear, plain-text advisory notices without polluting the interface with raw Python tracebacks. Furthermore, monitoring of completion metadata (finish_reason) was integrated to flag truncated or incomplete responses during background polling, ensuring absolute visibility over output fidelity.
Roadblocks, Traps & Avoiding Rabbit Holes
A disciplined development approach requires anticipating where engineering efforts can derail into unproductive complexity. Several major roadblocks were identified and systematically circumvented during the evolution of the dispatcher:
1. UI Token Bloat & Cross-Contamination
The Trap: Accidentally pulling historical file lists or global chat histories into current-turn API requests, causing massive token inflation and contaminating new prompts with stale context.
The Bypass: Restricting file-ID matching and document payload ingestion strictly to the active turn’s isolated <attached_files> XML boundaries, ignoring historical baggage entirely.
2. Fragile Disk Path Rebuilding
The Detailed Consideration: In early iterations, resolving uploaded attachments relied on reconstructing internal file paths dynamically using string manipulation or guessing where OpenWebUI stored uploads on the container volume. This approach is inherently brittle; any minor container update, change in storage hashing algorithms, or shift in directory naming conventions causes the lookup to fail silently or crash with a FileNotFoundError. Furthermore, hardcoding speculative directory assumptions invites cross-environment execution failures when migrating between local Docker instances and production deployments.
The Trap: Attempting to dynamically reconstruct file paths via internal container string guesses, which inevitably breaks when OpenWebUI updates its internal directory structure.
The Bypass: Abandoning path-guessing completely in favour of reading the absolute system paths provided natively within OpenWebUI’s runtime __files__ array dictionary objects. This ensures that the application interacts directly with verified, runtime-resolved file pointers rather than speculative guesses.
3. Scanned PDF False Positives
The Detailed Consideration: Ingesting PDF documents programmatically within a batch processing pipeline presents an insidious data-integrity challenge: image-only or poorly scanned PDFs containing no selectable text layer. If a script relies solely on checking whether a file exists or reading a flat, aggregate character count across the entire document, a multi-page scanned artifact containing just a few stray characters in a header or page number can easily slip past the threshold. This results in the model receiving a virtually blank payload, leading to hallucinations, wasted API tokens, and silent processing failures downstream.
The Trap: Relying on a flat total-character minimum threshold, which fails when a multi-page scanned document contains just enough stray header text to slip past unread.
The Bypass: Enforcing a strict proportional per-page density check (< 100 characters per page average) via pypdf extraction to reliably intercept and refuse image-only, un-OCR’d, or password-protected files upfront before any dispatch occurs.
4. Concurrency & Ledger Corruption
- The Detailed Consideration: Asynchronous batch dispatchers operate with background polling loops and recurring ledger updates running concurrently alongside user interactions. When a script writes directly to a local JSON tracking ledger mid-execution—especially while rapid status queries or new dispatches are occurring—an interruption or race condition can easily result in a half-written file. This leads directly to total JSON corruption, orphaned task IDs, and a complete loss of tracking state.
- The Trap: Writing directly to the local JSON ledger during rapid asynchronous polling intervals, risking half-written files and complete state corruption if a process terminates mid-write.
- The Bypass: Implementing a two-stage transactional pattern. All ledger updates are first written to a temporary staging file on disk, after which an atomic operating system swap (
os.replace) commits the change instantly, backed by thread locks to eliminate race conditions entirely.
Conclusion
As I always say, ‘A pilot does not need to know how every part of the plane works, in order to fly the blody plane!” The same applies to a HGV driver who may command hundred of horse power in his machine.
I don’t need to understand all the intricacies of how the technology works. Two powerful frontier models were harnessed to work together. The outcome is a solution that has not been found elsewhere on the internet.
Further testing and refinement will happen in the background on the way to near perfection.
