Build your own Shorts Scraper
Choose your niche. Study the right creators. Turn verified outliers into a private research library with playback, transcripts, metrics, and pattern analysis. Click one button when you want up to ten more.
Get one useful AI system every week
I share the workflows I actually use to run my content and company, including what stayed manual and what was worth automating.
You're in. First email lands shortly.
What this builds
I did not want another folder full of saved Reels I would never study. I wanted one place where I could play a video, read the transcript, see the captured engagement, inspect the hook and structure, and move to the next example without opening the feed.
The first version came from ten Reels I had already selected for a content pilot. I saved the media, corrected the transcripts, captured the visible metrics, inspected the videos, and joined those records into a private site. The site is useful because the evidence stays beside the analysis.
What it is: a research viewer over videos you collected and verified.
What it is not: a magic bot that automatically finds every viral post on Instagram and TikTok.
My current site reads a saved archive. The build in this guide adds one deliberate action: you click “Get up to 10 more” when you want a fresh set. Nothing runs each morning. Nothing sends a phone notification. The button has to show its work before a video reaches your library.
The outlier protocol
A high view count is not enough. The scraper compares each candidate with the creator who posted it. The threshold changes at 200,000 followers.
| Creator size | A video qualifies when | Required receipt |
|---|---|---|
| Under 200,000 followers | Views are at least 5× the creator’s follower count, or at least 5× the median views of the creator’s latest 10 videos. | Follower count and capture time. For the median route, save all 10 baseline view counts and the calculation. |
| 200,000 followers or more | Views are at least 2× the median views of the creator’s latest 10 videos. | All 10 baseline view counts, the sorted values, the median, and the candidate-to-median ratio. |
The candidate video stays out of its own baseline. A missing follower count or incomplete 10-video baseline produces an “insufficient evidence” label. It does not get surfaced as an outlier.
The one prompt to start with
If you want the AI to guide you through the whole build, paste this first. It forces the work through seven gates and makes the AI stop for your input instead of jumping straight into a half-built dashboard.
You are my Shorts Research Coach. Help me build a private, niche-specific short-form research library without inventing data or automating a process before it works. Work through seven gates in order. Ask no more than five questions at a time, wait for my answers, and tell me exactly what I need to do before you continue. 1. FOUNDATION: Interview me about my niche, audience, offer, platforms, regions, content formats, and what a useful research library should help me decide. 2. CREATOR SET: Help me choose 5 to 15 relevant creators and lock this outlier protocol. For creators under 200,000 followers, a candidate qualifies when its views are at least 5 times the creator's follower count or 5 times the median views of that creator's latest 10 videos. For creators with 200,000 followers or more, require the candidate to have at least 2 times the median views of that creator's latest 10 videos. Preserve the follower count, all 10 baseline view counts, the median calculation, the ratios, and the rule that qualified each result. 3. TOOL AUDIT: Inspect what tools and file access you actually have. Separate what you can do now from what needs a manual upload, a paid service, a login, or my approval. Never pretend you scraped a source you could not access. 4. EVIDENCE PACK: Help me collect up to 10 real qualifying videos. Preserve the creator, source URL, date captured, visible metrics and capture time, follower count, latest-10 baseline, median, calculated ratios, qualification rule, caption, local video file, and a missing-data note. Never estimate a metric. 5. INSPECTION: Transcribe every video, correct obvious errors, and inspect the opening, spoken hook, on-screen hook, visual hook, structure, angle, proof, CTA, editing format, pacing, and important frame changes. 6. SYNTHESIS: Compare the set. Separate repeated patterns from one-off choices. Only use the word outlier when the record passes the approved protocol and its calculations can be reproduced from the saved receipt. 7. SITE: Build a searchable private site with local playback, creator filters, source links, transcripts, captions, captured metrics, outlier calculations, analysis, previous and next navigation, and clear missing-data labels. Add a button labeled "Get up to 10 more." The button starts collection only when I click it, removes duplicates, applies the approved protocol, and returns up to 10 new qualifying videos. If only three qualify, return three. Show progress, cost, errors, and the last run time. Do not add automatic morning runs, email notifications, phone notifications, user accounts, or a database of users. Keep the first version single-owner and configuration-file based. Do not promise virality, leads, or hands-off operation. Respect platform terms and creators' rights. If a download is unavailable, ask me to provide the file. Start with Gate 1.
The rest of this guide breaks the same process into individual prompts. Use the long version when you want more control or need to rerun one stage.
The smallest tool stack
You can complete the manual proof without paying for a scraper. Use tools that inspect files and preserve sources. Add an API only after the ten-video workflow produces research you use.
| Job | Manual-first choice | Optional upgrade |
|---|---|---|
| AI operator | Claude Code or Codex | An always-on agent after the workflow is stable |
| Source collection | Your own files, creator-provided files, or a browser download you are allowed to make | A tested collector such as the Scrape Creators Instagram Posts API |
| Media inspection | FFmpeg and ffprobe | Cloud media processing when volume demands it |
| Transcription | Local Whisper | A paid transcript service if setup time matters more than cost |
| Research store | CSV or JSON plus a clear folder structure | A server-side store for video records after the schema stops changing |
| Website | A local static site | A private Vercel preview after the local version passes |
The Scrape Creators endpoint returns public Instagram posts and Reels for a handle. Its documentation warns that play counts sometimes differ from the combined number Instagram shows when a Reel was also shared to Facebook. Save the collection source beside every metric.
yt-dlp helps with sites it currently supports, but extractors break when platforms change. Treat it as a convenience, not a guaranteed Instagram or TikTok integration. If it cannot fetch a source, provide the video file yourself.
This version has one owner, one creator configuration, and no user-account database. API credentials stay in the hosting environment. They never reach browser code.
The seven-gate build
Run these in order, in one project folder and one conversation when possible. Each prompt tells the AI what evidence it needs, what it may not assume, and where it must stop for you.
Define the research job
You finish with: A one-page research brief that tells the AI what to study and what decisions the library should support.
We are starting Gate 1 of my Shorts Scraper build. Interview me before recommending creators or tools. Ask no more than five questions at a time and wait for each answer. You need to learn: - my niche and sub-niches - the audience I want to attract - the product, service, or next action my content supports - the platforms and regions I care about - the content formats I can realistically make - the result I want this research library to help me improve - the claims or topics I will not cover - my available time, budget, and comfort with technical setup Do not generate content ideas yet. Do not recommend creators yet. When the interview is complete, write a concise Research Brief with these headings: Audience, Business Goal, Content Scope, Formats, Platforms, Evidence Standard, Constraints, and Questions This Library Must Answer. Then ask me to approve or correct the brief before moving to Gate 2.
Choose creators and a fair sample
You finish with: A verified creator watchlist and a reproducible two-tier outlier protocol.
We are starting Gate 2. Use my approved Research Brief. Help me build a watchlist of 5 to 15 creators. First ask whether I already have saved posts, creator handles, or competitor research. If I do, inspect those before suggesting anyone new. For every creator, record: - exact display name and handle - platform - why this creator is relevant to my audience or format - the specific trait I want to study - whether the identity is verified - whether they are a direct competitor, adjacent creator, or format reference Reject accounts that are a poor audience match, inactive, misidentified, mostly sponsored, or impossible to verify. Use this exact outlier protocol: - Under 200,000 followers: a candidate qualifies when candidate views are at least 5 times the creator's follower count OR at least 5 times the median views of the creator's latest 10 videos. - At 200,000 followers or more: ignore the follower ratio and require candidate views to be at least 2 times the median views of the creator's latest 10 videos. For every creator, preserve the follower count and capture time. For every median test, preserve the 10 individual baseline view counts, sort them, and show the median calculation. Never include the candidate video in its own baseline. If the follower count, candidate views, or required 10-video baseline is unavailable, label the result insufficient evidence. The 5x and 2x thresholds are research filters, not virality guarantees. Return a Creator Watchlist table and a Collection Plan. Stop for my approval before Gate 3.
Audit tools and access
You finish with: A tool plan that distinguishes available capabilities from paid, blocked, or manual work.
We are starting Gate 3. Audit the tools and access available in this workspace before touching any social URL. Check for: - a file-capable AI agent such as Claude Code or Codex - a folder where source videos and records can live - FFmpeg and ffprobe for media inspection and frame extraction - a transcription option such as local Whisper - a browser or approved downloader for sources I am allowed to collect - a spreadsheet, CSV, or JSON store - a local web framework and a hosting option - an optional collector such as the Scrape Creators API or another source that passes a live test Create four lists: Available Now, Requires Installation, Requires Login or Payment, and Must Be Done Manually. Rules: - Do not ask me to paste passwords, cookies, or API keys into chat. - Do not install software, spend money, log in, upload files, or deploy without asking at the moment that action is needed. - A tool claiming support for Instagram or TikTok is not proof that it works today. Require a live test. - A downloader working is not permission to republish someone else's media. - If you cannot retrieve a video, ask me to supply the file. Do not fabricate a transcript or analysis. - Do not add user accounts, a user database, or a multi-tenant backend. This version has one owner and one editable creator configuration. Recommend the smallest manual stack that can complete one ten-video proof. Stop for approval before Gate 4.
Build the evidence pack
You finish with: Up to ten qualifying source receipts with every outlier calculation preserved.
We are starting Gate 4. Help me collect up to 10 qualifying videos from the approved creator set. Create one source receipt per video with these fields: - research ID - creator and verified handle - platform and source URL - date posted, if visible - date and time captured - views, likes, comments, shares, and saves exactly as visible - creator follower count and capture time - the view counts for the creator's latest 10 baseline videos when the protocol requires them - the sorted baseline values and calculated median - candidate views divided by follower count - candidate views divided by the latest-10 median - the exact qualification rule that passed - caption text - local video filename and file size - acquisition method - missing fields and why they are missing - rights or usage note Preserve the original files. Create smaller playback copies only when the site needs them. Never overwrite the archive. After each collection, verify that the URL, creator, file, duration, orientation, and visible metrics belong to the same post. Exclude the candidate from its own latest-10 baseline. Leave unavailable metrics blank. Do not convert blanks to zero. Reject records that do not pass the approved rule or cannot prove the calculation. Return a manifest of up to 10 qualifying receipts and list anything I still need to provide manually. Fewer than 10 is an honest result when fewer than 10 qualify. Do not analyze patterns yet. Stop before Gate 5.
Transcribe and inspect each video
You finish with: A grounded, frame-aware analysis for every video, tied back to the actual transcript and media.
We are starting Gate 5. Work through the ten verified source receipts one video at a time. For each video: 1. Transcribe it with timestamps. 2. Correct obvious transcription errors only when the audio or on-screen text supports the correction. 3. Extract representative frames at the opening and every meaningful visual change. 4. Record the exact spoken hook, on-screen text hook, and visual hook. 5. Map the structure beat by beat. 6. Identify the topic, angle, promise, value, proof, CTA, editing format, editing style, pacing, pattern interrupts, and ending. 7. Separate observation from interpretation. Quote only short phrases needed to support the analysis. 8. Mark anything you could not inspect as unknown. Do not copy a creator's whole script or identity. Focus on reusable decisions such as a hook shape, sequence, proof placement, or visual format. Return one standardized analysis record per video. Stop after all ten pass a source-to-analysis check.
Find repeatable patterns
You finish with: A pattern report that distinguishes evidence across the set from attractive one-off choices.
We are starting Gate 6. Compare only the ten verified analyses and the approved Research Brief. Build a Pattern Report with: - hook patterns that repeat - common structures and their order - recurring angles - proof types and where proof appears - CTA patterns - editing formats and visual changes - pacing ranges - audience problems and desires that recur - patterns that fit my realistic production constraints - one-off choices that should not be treated as rules - missing evidence that prevents a conclusion For every conclusion, cite the research IDs that support it. Separate Frequency, Possible Function, and Confidence. Do not claim causation from views alone. Then produce a Modeling Menu. For each promising pattern, state what can be borrowed ethically, what must come from my own experience, and what would make the result too close to the source. Do not generate finished content ideas unless I ask after reviewing the report. Stop for my corrections before Gate 7.
Build and verify the private site
You finish with: A searchable single-owner research site with an on-demand button for up to ten more verified outliers.
We are starting Gate 7. Build the smallest private research site that makes the approved evidence easy to study. Required experience: - creator filter - search across hooks, transcripts, captions, and structures - vertical local video playback - transcript, caption, and analysis tabs - captured metrics with the capture date - follower count, latest-10 median, calculated ratios, and qualification rule - original source link - previous and next navigation - a visible research ID - explicit unknown or insufficient baseline labels - a button labeled "Get up to 10 more" - progress, last-run time, records checked, records accepted, records rejected, cost, and errors for every button run Use my brand colors and fonts, but prioritize legibility. Do not copy another site's logo, product name, or exact visual identity. Do not expose private files, credentials, cookies, or local paths. The button is the only collection trigger. When I click it, collect a fresh candidate pool from the approved creator list, deduplicate by platform and post ID or canonical URL, apply the approved two-tier outlier protocol, and add up to 10 new qualifying records. Never invent filler to reach 10. Do not add automatic schedules, morning notifications, email alerts, phone alerts, user accounts, or a database of users. Store one owner's editable creator list and settings in a simple configuration file. Keep API secrets in the hosting environment, never in browser code or the repository. Build from the verified manifest instead of hand-entering records twice. Add checks that fail when a video, transcript, receipt, or required analysis field is missing. Verify all ten records, mobile layout, keyboard access, local media playback, copy buttons, source links, and search. Deploy to a private preview or keep it local according to my approved plan. Report what is real, what is static, and what is still manual. Do not describe the button as working until one live click retrieves, filters, saves, and displays at least one verified record.
Add the “Get up to 10 more” button
Once the manual run works, add one intentional collection action. You decide when the system spends requests, pulls a candidate pool, and adds new records. The run reports how many candidates it checked and why each rejected video failed.
The button returns up to ten verified outliers. If three qualify, it returns three. It never weakens the rules or adds filler to make the grid look full.
The real button connects to a server-side collector. API credentials never enter the browser.
- Last run
- Not run
- Checked
- 0
- Accepted
- 0
- Rejected
- 0
- Request cost
- $0.00
The manual proof is complete. Add one on-demand collection action to the private Shorts Scraper site. Create a button labeled "Get up to 10 more." Do not create a schedule, email alert, phone notification, user account system, or user database. When I click the button: 1. Read the approved creator list from one editable configuration file. 2. Retrieve a fresh candidate pool through the collector we already tested. Keep the API key in the hosting environment. 3. Deduplicate by platform plus post ID or canonical URL before spending time on transcription or frame analysis. 4. Apply the approved protocol. Under 200,000 followers, require either 5 times the follower count or 5 times the latest-10 median. At 200,000 followers or more, require 2 times the latest-10 median. 5. Preserve the follower count, capture time, 10 baseline view counts, median calculation, ratios, and qualification rule for every accepted result. 6. Add up to 10 new qualifying videos. Return fewer when fewer qualify. Never add filler. 7. Show progress, last-run time, candidates checked, accepted count, rejection reasons, request cost, and errors. Test the complete click once with one approved creator. Compare every returned field with the visible source. Do not label the button working until a real click retrieves, filters, saves, and displays at least one verified record. Never estimate missing metrics or promise virality, leads, or hands-off operation.
When the first version is done
- Ten videos play from local or controlled files.
- Every record has a creator, canonical source URL, and capture date.
- Visible metrics match the source receipt. Missing values stay blank.
- Every transcript was checked against the audio or on-screen text.
- Every analysis separates observation from interpretation.
- Under 200,000 followers, each surfaced video passes a 5× follower or 5× latest-10 median test.
- At 200,000 followers or more, each surfaced video passes the 2× latest-10 median test.
- The candidate is excluded from its own baseline and every median can be reproduced.
- Search finds hooks, captions, transcripts, structures, and angles.
- “Get up to 10 more” reports candidates checked, accepted, rejected, cost, errors, and last-run time.
- No morning schedule, notifications, user accounts, or user database were added.
- Private files, credentials, cookies, and local paths are not exposed.
- You used the library to make one real decision before expanding it.
Make the resource cover
If you want to package your own version as a guide or internal playbook, paste this into your image generator. The prompt avoids fake dashboards and uses a simple research workflow instead.
Create a wide banner image in 2:1 aspect ratio, 1600 by 800 pixels. Background: rich violet #7C3AED with a subtle radial glow and restrained navy #0F1729 shadow shapes. Text: "BUILD YOUR OWN SHORTS SCRAPER" in bold white geometric sans-serif similar to Plus Jakarta Sans. All caps. Break it across two lines after "OWN". Illustration: an abstract white-line workflow showing a vertical phone with a play symbol flowing into organized transcript cards, then a magnifying glass, a folder, and a simple analytics line. Add small electric-blue #2563EB accents. Style: clean, premium, modern, approachable, vector-like editorial illustration. Wide layout with the title on the left and the workflow on the right. Do not include people, faces, creator handles, social-platform logos, metrics, screenshots, subtitles, bylines, watermarks, or any text other than the exact title.