web-crawler
Technical SEOstarchild-ai-agent/official-skillsskills.sh ↗
Installs
3,858
deduplicated, at the last sync
Since we started
+0.2%
6 readings, about 15 hours apart. Not a live curve.
Our category
Technical SEO
ours
Last read
Sep 3, 2026
from the directory
Our brief
oursThe web-crawler is an advanced scraping tool designed to extract public data from diverse sources, including social media platforms like YouTube, TikTok, Instagram, and Reddit [1]. It specializes in handling complex extraction tasks such as downloading transcripts or scraping content from JS-heavy pages that are blocked by anti-bot measures (e.g., Cloudflare) [1].
- Markdown formatted article text/content
- YouTube video metadata and transcript
- Social media post data (profiles, posts)
- Web archive snapshots from paywalled sites
- Structured data from Chinese apps via Apify Store
- Python runtime environment
- Target URLs or search parameters
not detected
The evidence details that certain functions, particularly those using the Apify store, operate on a pay-per-event billing system based on credits [1].
not detected
The evidence does not contain information regarding user registration requirements.
The skill cannot verify if an archive snapshot exists; if the resulting markdown is empty, no snapshot was saved previously [1]. It also cannot fabricate article content or data when a required snapshot or source is unavailable [1].
Evidencestatic findingsweb-crawler/SKILL.md:1-242
[{"code":"env_var","match":"SCRAPECREATORS_API_KEY","path":"SKILL.md"},{"code":"env_var","match":"FIRECRAWL_API_KEY","path":"SKILL.md"},{"code":"env_var","match":"APIFY_TOKEN","path":"SKILL.md"},{"code":"api_key_mention","match":"API key","path":"SKILL.md"},{"code":"api_key_mention","match":"api-key","path":"SKILL.md"},{"code":"payment","match":"Billing","path":"SKILL.md"},{"code":"payment","match":"billing","path":"SKILL.md"},{"code":"payment","match":"pricing","path":"SKILL.md"},{"code":"endpoint","match":"https://docs.scrapecreators.com/v1/tiktok/profile/openapi.json`","path":"SKILL.md"},{"code":"endpoint","match":"https://api.scrapecreators.com/v1/youtube/video/transcript","path":"SKILL.md"},{"code":"endpoint","match":"https://api.scrapecreators.com/v1/tiktok/profile?handle=charlidamelio","path":"SKILL.md"},{"code":"endpoint","match":"https://api.firecrawl.dev/v2/scrape","path":"SKILL.md"}]---
name: web-crawler
version: 2.6.1
description: 'Web scraping plus social data: YouTube, TikTok, Instagram, LinkedIn,
Reddit, Threads, plus robust web-page fallback extraction.
Use when extracting public posts, transcripts, or pages JS-heavy enough to block
plain fetch (e.g. download YouTube transcript, scrape TikTok comments, listing pages
behind anti-bot/Cloudflare). Auto-fallback trigger terms: web_fetch failed, 403,
anti-bot, cloudflare, JS-heavy page.
'
metadata:
starchild:
emoji: "\U0001F577\uFE0F"
skillKey: web-crawler
requires:
bins:
- python
tags:
- scraping
- social-media
- web-crawler
- tiktok
- instagram
- youtube
- linkedin
- facebook
- twitter
- reddit
- threads
- bluesky
- pinterest
- snapchat
- twitch
- truth-social
- tiktok-shop
- google
- ad-library
- creator-data
- transcripts
- trending
- web_fetch-failed
- cloudflare
- anti-bot
- js-heavy
- listing-page
- fallback
- "\u53CD\u722C"
- "\u6293\u53D6\u5931\u8D25"
- "\u81EA\u52A8\u56DE\u9000"
user-invocable: true
disable-model-invocation: false
---Read 2 of 2 text files in the skill.
Installfrom the directory
npx skills add https://github.com/starchild-ai-agent/official-skillsInstalling happens there, not here. We are an index with an opinion, not a mirror.
What is inside itfrom the directory
3 files — names only. The directory does not report sizes.
What the auditors foundfrom the directory
A skill is instructions your agent will follow and scripts it may run, so who checked it matters as much as how many people installed it.
Installs, reading by readingours
Axis starts at 3.9k, not zero — the range is 3.9k to 3.9k.