web-crawler

Technical SEO

starchild-ai-agent/official-skillsskills.sh ↗

Installs

3,858

deduplicated, at the last sync

Since we started

+0.2%

6 readings, about 15 hours apart. Not a live curve.

Our category

Technical SEO

ours

Last read

Sep 3, 2026

from the directory

Our brief

ours

The web-crawler is an advanced scraping tool designed to extract public data from diverse sources, including social media platforms like YouTube, TikTok, Instagram, and Reddit [1]. It specializes in handling complex extraction tasks such as downloading transcripts or scraping content from JS-heavy pages that are blocked by anti-bot measures (e.g., Cloudflare) [1].

What it produces
  • Markdown formatted article text/content
  • YouTube video metadata and transcript
  • Social media post data (profiles, posts)
  • Web archive snapshots from paywalled sites
  • Structured data from Chinese apps via Apify Store
What it needs
  • Python runtime environment
  • Target URLs or search parameters
Paid services

not detected

The evidence details that certain functions, particularly those using the Apify store, operate on a pay-per-event billing system based on credits [1].

Registration

not detected

The evidence does not contain information regarding user registration requirements.

Limits and human review

The skill cannot verify if an archive snapshot exists; if the resulting markdown is empty, no snapshot was saved previously [1]. It also cannot fabricate article content or data when a required snapshot or source is unavailable [1].

Evidencestatic findingsweb-crawler/SKILL.md:1-242
static findingsstatic_finding
[{"code":"env_var","match":"SCRAPECREATORS_API_KEY","path":"SKILL.md"},{"code":"env_var","match":"FIRECRAWL_API_KEY","path":"SKILL.md"},{"code":"env_var","match":"APIFY_TOKEN","path":"SKILL.md"},{"code":"api_key_mention","match":"API key","path":"SKILL.md"},{"code":"api_key_mention","match":"api-key","path":"SKILL.md"},{"code":"payment","match":"Billing","path":"SKILL.md"},{"code":"payment","match":"billing","path":"SKILL.md"},{"code":"payment","match":"pricing","path":"SKILL.md"},{"code":"endpoint","match":"https://docs.scrapecreators.com/v1/tiktok/profile/openapi.json`","path":"SKILL.md"},{"code":"endpoint","match":"https://api.scrapecreators.com/v1/youtube/video/transcript","path":"SKILL.md"},{"code":"endpoint","match":"https://api.scrapecreators.com/v1/tiktok/profile?handle=charlidamelio","path":"SKILL.md"},{"code":"endpoint","match":"https://api.firecrawl.dev/v2/scrape","path":"SKILL.md"}]
web-crawler/SKILL.md1–242 · excerpt truncated
---
name: web-crawler
version: 2.6.1
description: 'Web scraping plus social data: YouTube, TikTok, Instagram, LinkedIn,
  Reddit, Threads, plus robust web-page fallback extraction.


  Use when extracting public posts, transcripts, or pages JS-heavy enough to block
  plain fetch (e.g. download YouTube transcript, scrape TikTok comments, listing pages
  behind anti-bot/Cloudflare). Auto-fallback trigger terms: web_fetch failed, 403,
  anti-bot, cloudflare, JS-heavy page.

  '
metadata:
  starchild:
    emoji: "\U0001F577\uFE0F"
    skillKey: web-crawler
    requires:
      bins:
      - python
    tags:
    - scraping
    - social-media
    - web-crawler
    - tiktok
    - instagram
    - youtube
    - linkedin
    - facebook
    - twitter
    - reddit
    - threads
    - bluesky
    - pinterest
    - snapchat
    - twitch
    - truth-social
    - tiktok-shop
    - google
    - ad-library
    - creator-data
    - transcripts
    - trending
    - web_fetch-failed
    - cloudflare
    - anti-bot
    - js-heavy
    - listing-page
    - fallback
    - "\u53CD\u722C"
    - "\u6293\u53D6\u5931\u8D25"
    - "\u81EA\u52A8\u56DE\u9000"
user-invocable: true
disable-model-invocation: false
---

Read 2 of 2 text files in the skill.

Checked August 2026 · GemmaWritten from the skill's published files. Check the source before relying on access or cost details.

Installfrom the directory

npx skills add https://github.com/starchild-ai-agent/official-skills

Installing happens there, not here. We are an index with an opinion, not a mirror.

What is inside itfrom the directory

exports.py
reference/apify-pricing.md
SKILL.md

3 files — names only. The directory does not report sizes.

What the auditors foundfrom the directory

A skill is instructions your agent will follow and scripts it may run, so who checked it matters as much as how many people installed it.

Gen Agent Trust HubThe skill is a web crawler designed to extract data from social media platforms and websites using external APIs. The primary security consideration is the risk of indirect prompt injection, as the agent processes untrusted content from the internet which could contain malicious instructions. No evidence of credential theft, unauthorized data exfiltration, or malicious code execution was found.May 23, 2026 · SAFEpass
Socket1 alert: gptAnomalyMay 23, 2026warn
SnykRisk: MEDIUM · 1 issueMay 23, 2026 · MEDIUMwarn

Installs, reading by readingours

3.9k
3.9k
Aug 30, 20266 readings over 4 days, drawn as the last reading of each day.Sep 3, 2026

Axis starts at 3.9k, not zero — the range is 3.9k to 3.9k.

Filed alongside itours