Building the Eros Now Content Pipeline: Start to End
I’ve mentioned before that I worked on multimodal AI stuff at Xfinite Global PLC (erosnow.com), but I kept that post pretty vague on purpose. This is the real version — about nine months, August 2024 through May 2025, going from a folder of random one-off notebooks to an actual GPU-aware pipeline running in production, plus what happened after that. Including the parts that didn’t work, not just the highlight reel.
the actual problem
Eros Now has a big catalog of Indian cinema. Before something goes up — or an old title needs re-checking — someone has to answer: does this need a compliance flag (nudity, violence, smoking, whatever), where do scenes actually start/end, what’s happening in each one, what’s being said. You can’t do that by hand at this scale. That was my job — build the automated version of that review.
day one, three different APIs
The git history pins this down better than my memory would: on November 8, 2024, I tried three completely different Google Cloud content-analysis APIs, same day, in parallel — not because I had a plan, but because I genuinely didn’t know which one would actually be good enough yet.
- Cloud Vision API — SafeSearch, labels, object localization
- Video Intelligence API — explicit-content detection, labels, speech transcription
- Gemini Flash (OCR) — the one that eventually won out
The Video Intelligence one gave me the cleanest “lesson learned in real time” moment of this whole project. First attempt, I passed a local file path straight in as the input — except that API wants a gs:// URI, not a local path, so it just failed immediately. Next cell in the same notebook: switch to passing raw file bytes instead for local files, and it works, correctly finding elephant/jungle/wildlife tags on a test clip. A bit later, back to a proper gs:// path, and speech transcription with word-level timestamps works fine too. One notebook, one dumb mistake, one fix, all captured in order.
The Vision API side had its own rough edges — a copy-paste bug where a print statement mislabels the smoking-likelihood as “racy,” and I ended up manually rate-limiting requests with a sleep(5) between them because the classic API’s quotas were annoyingly slow. Got through about 23 frames before hitting a KeyboardInterrupt and just calling it there.
Gemini Flash won. Everything after this is built on it.
Also that same week: a face-recognition attempt (I grabbed reference photos of a well-known Bollywood actor to try face tracking — that’s literally as far as it got, no recognition code was ever written against it), plus PySceneDetect-based shot detection as a first, simple pass at scene boundaries, plus OCR extraction and a couple of throwaway format-conversion notebooks. None of that was wasted time — that’s how you figure out what’s actually worth building on.
turns out I copy-pasted one scratchpad everywhere
Going back through every notebook line by line for this post, I found something I honestly didn’t fully realize at the time: basically the entire gemini_flash-inference/ folder is one boilerplate template, copy-pasted over and over — same initialize_vertex_ai, get_generation_config, get_safety_settings, load_text_from_file, process_images, main() functions, reused across OCR, object detection, brand detection, scene description, compliance, captions, transcription, all of it.
I’m not guessing on this — I actually checksummed the files. Entire post-processing scripts are byte-for-byte identical across three unrelated experiment folders. Even a random empty file I must’ve accidentally created with touch shows up byte-identical in all three. One notebook’s checkpoint is literally a saved copy of a completely different experiment’s notebook, just sitting under the wrong folder name.
Is that a little sloppy? Sure. But it’s also just what fast iteration actually looks like — once “send Gemini some frames with a prompt and get JSON back” was solved once, every new idea was a copy of that with a new prompt, not a new pipeline from scratch. The actual thinking moved from “how do I call this model” to “what do I ask it, and can I trust what it hands back.”
It went further than the boilerplate code, too: the scene-description experiment’s system prompt is a byte-for-byte, MD5-verified copy of the brand-detection prompt — still describing “brand detection” while living in the scenes folder. The actual per-experiment prompt.txt was correctly rewritten for scenes, so it still worked, but the system instruction was never updated. That’s the copy-paste habit showing up in the actual prompts, not just the surrounding code.
Once the boilerplate was solved, every new experiment was really just: new prompt, new post-processing script.
- Object detection — told the model to act “like a strict CNN-based object detector, similar to Detectron and YOLO,” and separately tried a fixed ~500-item vocabulary as an alternative to just letting it detect anything, to see if anchoring it to known labels helped.
- Brand/logo detection — this one actually worked well. Real production-company logos from a movie’s opening credits, real advertising brands in the background. Eventually turned into a proper script that at least counts total errors across the whole run and warns if it crosses 50 — not a real circuit breaker (it only checks after everything’s already finished processing, doesn’t actually stop anything early), but still better than silently not knowing something went wrong.
- Scene/location/event description — batched 30 frames per call instead of one at a time, and got solid scene-by-scene descriptions out of it. The location experiment had a nice small design choice: instead of the model saying “uncertain,” I had it append “(probably)” to low-confidence guesses and just say “unidentified” when it had nothing — small thing, but it made filtering the output way easier later, and it worked exactly as intended.
- Captioning — I told it to caption “in the language of the audio.” It didn’t listen. Captions came back in English prose even on Hindi-audio clips. That’s just a real limitation I hit, not a guess.
- Transcription — the prompt and the system instruction for this one actually tell two different stories: one’s framed around generating subtitles, the other’s framed as strict “just transcribe the audio, ignore everything else.” Going back through the commit history, both landed together in the same commit, not as a rewrite over time — just two framings that never quite got reconciled with each other.
A few real bugs showed up along the way: the model would sometimes return almost-valid JSON with a trailing comma before a closing bracket, patched early on with a regex cleanup pass; Gemini’s safety filters blocked requests mid-batch more than once (a PROHIBITED_CONTENT block on completely ordinary movie frames), which is why a lot of the batch scripts track a running list of blocked images instead of crashing; and one location-detection cell had a broken regex that matched literally zero results, on every frame, silently wrote an empty output file, and got re-run a second time without anyone noticing.
the two ideas that actually stuck: compliance detection and video splitting
Out of everything above, two ideas survived and turned into the real backbone — and it’s worth walking through how much each one changed shape, because none of it happened in one go.
Gemini-based compliance detection. Generic, Western-trained content moderation just doesn’t work on Indian cinema — a saree isn’t nudity, and a classifier trained mostly on Western norms will happily flag it as one. So the system prompt explicitly carves out traditional clothing, defines twelve fixed compliance categories, and forces a confidence score per detection instead of a flat yes/no.
The output format changed mid-project, not just the prompt. An early version always returned all twelve category keys, mostly empty, plus a free-text comment. What’s actually documented now is different — only the categories it actually found, each with its own score and note. The model and settings changed too, mid-run: an earlier checkpoint used Gemini 1.5 Flash at temperature 0.4; the final version moved to Gemini 2.0 Flash at 0.05, once the goal shifted from “let’s see what it notices” to “give me the same answer every time.”
And then it became a real service, months later, not right away. The notebook versions are from December 2024 and January 2025. The actual production script — a proper CLI with argparse, real logging, thread-pool parallelism, retries, and checkpointed intermediate saves so a crash doesn’t lose everything — didn’t land until April 2025. That script also finally killed the trailing-comma JSON bug for good, by turning on Gemini’s structured JSON output instead of regex-patching bad responses after the fact.
Video splitting, built on Panda-70M, not invented by me. This is the clearest example in the whole project of standing on someone else’s work instead of reinventing it. The video-splitting piece is, plainly, an adapted clone of Snap Research’s Panda-70M splitting pipeline — I found the original, unedited Panda-70M README, with the original paper author’s email still in it, sitting in a leftover Jupyter checkpoint file. Two of the three stages — shot detection with PySceneDetect, and semantic stitching with Meta’s ImageBind — are byte-for-byte identical to their code. The one thing I actually changed was renaming output clips from their default numeric names to human-readable timestamps.
The pipeline itself: PySceneDetect finds hard cuts, ImageBind checks how semantically similar the start/end frames of neighboring shots are to decide if they’re really the same event, then ffmpeg does the actual cutting. What was actually mine here was the tuning work — shell scripts running the stitching/splitting stages across five real movies with different threshold setups (“Ideal,” “IdealMax,” “Max”), logging system resource use, running several in parallel with GNU parallel — figuring out empirically what thresholds actually worked for this catalog instead of trusting Panda-70M’s defaults blindly.
The lesson from both: the hard part was rarely the thing that looked hard at first. For compliance detection, “can Gemini describe a frame” was trivial — the actual work was the schema and the crash-safe execution. For splitting, “can I detect shot boundaries” was already solved research — the work was finding the right thresholds empirically instead of guessing.
two audio pipelines running side by side
For audio (VAD, speaker diarization, transcription) I didn’t just pick a stack and go. I built two, in parallel: one on SpeechBrain, one on Whisper. Only one of them is actually in good shape.
The SpeechBrain pipeline does VAD, then ECAPA-TDNN speaker embeddings, then clusters speakers with agglomerative clustering. What’s not standard is that I left behind four separate self-written documents tracking the gap between what I planned, what I actually built, and how it compared to SpeechBrain’s own official recipe. One planning doc is basically a wishlist of stub classes that mostly never got built. The doc that actually matters is the one where I wrote down, plainly: “VAD threshold (0.1) is too low for Bollywood movies with background music” — checked against the real config file for this post, and 0.1 is still the real value. Same doc flags the speaker-similarity threshold as questionable, the fixed 0.5-second chunk duration as too short, and lists things I knew were missing and never got to: Hindi-specific models, separating music from speech, emotion detection.
There’s also a document comparing my implementation line-by-line against SpeechBrain’s official VoxCeleb recipe, including a comment I left myself flagging that actual speaker-verification logic was never implemented. And here’s the part that stings: the test file for this pipeline imports a class name that doesn’t exist anywhere in the actual module — it would fail immediately on import. I found this going back through the code for this post, not at the time, which says a lot about how “documented but abandoned mid-iteration” this pipeline really was.
For Whisper I tried two setups: HuggingFace Transformers’ Whisper large-v3 with flash-attention and forced Hindi decoding, and separately faster-whisper’s large-v3-turbo with a CPU-then-CUDA fallback (the CPU-first load is a real, deliberate workaround for a CUDNN initialization issue, not a stylistic choice). The two most feature-complete files on the Whisper side — the ones combining diarization with transcription — both, as it turns out, have actual syntax errors in them right now: one has a duplicated, half-pasted code block, the other has a stray semicolon sitting between the imports and the logging setup. Neither would run as committed. I only found this rerunning everything for this post.
So: neither audio pipeline was production-ready, for different reasons. SpeechBrain has real thought put into diagnosing its own gaps but the tests don’t even import correctly. Whisper’s simpler scripts are clean and work, but its two most ambitious files are currently broken by small, dumb syntax mistakes a five-second lint pass would’ve caught.
turning it into one real pipeline
By April 2025 everything above got pulled into one real package: download a movie, split it into clips, upload them, run Gemini compliance detection, upload the results. Every movie moves through defined stages — init, download, process, upload clips, compliance detection, upload results, cleanup, done — and each stage only runs if the previous one actually finished, so a crash resumes instead of restarting.
It has real multi-user support too — separate state files, namespaced storage paths per user/job — written specifically in anticipation of eventually putting a FastAPI layer in front of it. And this isn’t theoretical: the actual pipeline state file shows 131 movies fully processed end-to-end, download through compliance detection, over about five days in April 2025.
A decent chunk of the pipeline exists because GPUs on a shared training box aren’t always available. There’s a documented before/after tuning change: I dropped the GPU check interval from 30 seconds to 5, and the stability period from 60 to 30, specifically so the pipeline would resume faster once a GPU freed up. There’s also a fallback GPU-check script that talks to nvidia-smi directly instead of importing anything from the rest of the codebase, built after hitting real ModuleNotFoundError issues in production — and that’s actually the version the pipeline uses today, not the fancier one.
Here’s the part I have to admit: there’s a module the main pipeline code imports — gpu_utils.py — that flat out doesn’t exist in the repo, and never has, according to git history. Four different files reference it and would fail to import as committed. What saves this from being a real embarrassment: the actual production script doesn’t use any of those four files — it calls the standalone nvidia-smi fallback directly, which doesn’t need the missing module at all. So production never hit the gap, but it’s a real loose end I’m writing down instead of quietly fixing and pretending it was never broken. There’s also a smaller bug in the state-management CLI where datetime is only imported inside the if __name__ == "__main__" block, but used by a function that could be called separately — it would crash if anyone ever imports that module instead of running it directly.
a POC I built once and never touched again
Separate from all of the above — while exploring agent-style workflows, I went through a full walkthrough of Google’s LangChain + Vertex AI Reasoning Engine sample. Built a small agent with a currency-exchange tool, ran it locally, deployed it as an actual hosted Reasoning Engine resource on Vertex AI, queried the deployed version, and cleaned it up afterward. It’s one of the only notebooks in this entire project that ran start to finish without leaving an error behind — and then I never used it again. Nothing else in the project references Reasoning Engine or that agent pattern. It was purely a “let me understand how this works” exercise. (There’s also a separate, unrelated two-cell notebook in the same folder that never went past pip install langchain-core langchain-google-vertexai — a completely different dead end, not this one.)
giving the pipeline an actual API and a frontend
I didn’t build a production FastAPI + LangGraph service from scratch for this. I started from fastapi-langgraph-agent-production-ready-template, an open-source project by Wassim EL BAKKOURI, with real contributions from a couple of other people too. The JWT auth, session management, rate limiting, Prometheus/Grafana monitoring, the evals framework, the Docker setup, the base LangGraph agent scaffold — none of that is mine. Using it instead of reinventing auth and rate limiting for the tenth time was the right call.
What’s actually mine is four commits on top of that fork: switching the agent to use Vertex AI instead of its original LLM provider, and one big one — a full video-processing feature (56 files) plus an entire Next.js frontend, built from nothing. The connection to the movie pipeline above is real and literal: the adapter code does from movie_processing.pipeline import MoviePipeline, importing directly from the pipeline described above by appending its path to Python’s import path at runtime, with the config pointing at the exact same GCS bucket and GCP project ID. This is, concretely, the FastAPI layer that the pipeline’s own multi-user documentation said it was “ready for” — built in a separate repo, on top of someone else’s agent template.
On top of the fork: async job execution (the adapter runs the pipeline inside a thread-pool executor so a multi-hour job doesn’t block the API), real database models for the job lifecycle (video, processing job, clip, event log), a storage service wrapping GCS uploads and signed URLs behind the existing auth, a REST surface for upload/process/list/status, and a full Next.js frontend — login/register, a dashboard, a chat interface with SSE streaming.
Building the pipeline itself is one kind of work: get the models to do the right thing, make a batch job survive crashes, handle GPUs being unavailable. Wrapping it in this template is a different kind of work: it has to be safe for more than one authenticated user, async so a slow job doesn’t tie up a request thread, and observable enough that someone using a frontend knows what’s happening instead of staring at a spinner. That’s closer to normal product/backend engineering than to ML engineering.
If you look at that repo’s commit graph, most of the commits aren’t mine — they’re the original template plus a couple of merged community PRs. Mine are the Vertex AI switch and the video-processing/frontend commit. Both things are true at once.
so what actually happened here
The honest version of this story isn’t “I designed a multimodal pipeline and built it.” It’s: I tried a bunch of small things, most of which went nowhere, kept the two that actually worked, ran two competing audio approaches before picking a direction, converged all of that into one GPU-aware, multi-user pipeline that actually ran against 131 real movies, found a couple of real loose ends only while writing this up, built one unrelated POC purely to learn something, and eventually wrapped the whole thing in a real API and frontend built on top of someone else’s open-source template. That’s a much messier story than “I built a multimodal AI pipeline,” but it’s the actually true one.
Tech Stack: Python, Gemini (Vertex AI), Google Cloud Storage, ImageBind, PySceneDetect, ffmpeg, SpeechBrain, Whisper / faster-whisper, PyTorch, GCP GPUs (Tesla T4 for local splitting, A100/L4 for Vertex AI inference), LangChain, Vertex AI Reasoning Engine, FastAPI, LangGraph, Next.js, PostgreSQL, Docker
If you’re building something similar and want to compare notes on any of this — the audio-stack decision especially — reach out.