v1.5 · open source · MIT

Tell it what you want.
It finds, downloads,
reads & files it.

An AI scraping CLI that searches the web, keeps only the file types you ask for — PDFs, docs, slides, anything — pulls them in parallel (Scribd and Google Drive too), OCRs the text and drops it all into tidy folders.

See the benchmark →
scrap — zsh

    

  


  
  

    
0× faster than a
plain download loop

    
0file-type presets
+ any extension

    
0byte-range chunks
per file, 12 files at once

    
0API keys required
(Ollama / no-AI mode)

  


  
  

    

      
01 — Scrape only what you need

      
Not every page. Not every file.
Just the ones you asked for.

      
Pick types and watch the command build itself. Scribd and Drive are crawled only when you pick documents.

    

    

      

        
        
        
        
        
        
        
        
        
        
        
      

      
$ 

      
    
    
    
  
    

  
  
    
    
    
      
    02 — Benchmark
    
      
    Same bytes. Same network.
    One finishes first.
    
      
    A local server caps each connection at 2 MB/s with 60 ms latency. Plain tools get a hand-filtered list; MANOLO Scrap gets the full mixed list and picks the right files itself. Reproduce it with scrap-master bench.
    
    
    
    
    
      
    MANOLO Scrap
    0.23s100
    
      
    curl -Z (parallel)
    1.09s23
    
      
    python loop
    3.98s13
    
      
    curl (sequential)
    4.05s13
    
      
    
    
    
    
      
    60%Speed vs fastest
    
      
    20%Filters types itself
    
      
    10%Files into folders
    
      
    10%Byte-exact (sha256)
    
    
    
  
    

  
  
    
    
    
      
    03 — How it works
    
      
    One sentence in.
    An organized library out.
    
    
    
    
      
      
    1. 1
      Plan
      Ollama or OpenRouter turns your request into search queries, file types and depth. No AI? A keyword plan takes over.
      2. 
      
    2. 2
      Search & crawl
      Keyless search across four backends, then same-site crawling for linked documents.
      3. 
      
    3. 3
      Filter
      Only the types you picked survive. Scribd and Drive links join when documents are wanted.
      4. 
      
    4. 4
      Download
      12 files at once, each split into 8 HTTP Range chunks, with retries and PDF checks.
      5. 
      
    5. 5
      Read & file
      PyMuPDF + OCR to Markdown, files sorted by type, and an AI report.md.
      6. 
    
    

    
    
      
    ~/MANOLO-Scrap/projects/2026-10-16_form-4-chemistry/
├── files/
│   ├── pdf/        past-paper-2024.pdf  marking-scheme.pdf
│   ├── slides/     revision-deck.pptx
│   └── sheets/     grade-boundaries.xlsx
├── scribd/      scribd_843737831.md
├── markdown/    past-paper-2024.md   (OCR)
├── pages/       every crawled page as Markdown
├── data/        plan.json  search_results.json  harvest_manifest.json
└── report.md    AI summary of what was collected
    
    
    
  
    

  
  
    
    
    
      
    Scribd → Markdown
    Full documents extracted straight to clean Markdown, no viewer screenshots.
    
      
    TUI included
    Type scrap for a keyboard-first app: Ask AI, tools, file viewer, health check.
    
      
    Sort any folder
    scrap-master organize ~/Downloads files everything by type in one pass.
    
      
    Agent-ready
    Ships an MCP server and an agent skill so Claude and other agents can scrape for you.
    
    
    
  
    

  
  
    
    
    04 — FAQ
    Questions
    
    
    
      
    Can it download only PDFs?
    Yes — --only pdf, or write “pdfs only” in your request. Mix presets like --only docs,slides or use any extension, e.g. --only .djvu.
    
      
    Does it work with Scribd and Google Drive?
    Yes. Scribd documents become Markdown; Drive files are exported directly. Turn either off with --no-scribd / --no-drive.
    
      
    How is it faster than curl or wget?
    It downloads many files concurrently and splits each file into byte-range chunks, so per-connection speed caps stop mattering. Run scrap-master bench on your machine to check.
    
      
    Do I need a paid AI key?
    No. Use a local Ollama model, an OpenRouter free model, or --no-ai.
    
    
    
  
    

  
    
    
    Stop clicking “download” forty times.