▄▀▄ ▄▀▄
█▄▀ █▀█
▀ ▀ ▀ ▀ by webXOS
Uses a "Datahawk Engine" - a browser-native synthetic data factory with a 15-wing fanned API engine and the OpenWiki pipeline. Fans across free research APIs and live-data APIs (Wikipedia, DuckDuckGo, arXiv, OpenAlex, Crossref, Semantic Scholar, GitHub, Open-Meteo weather, REST Countries, Open Library, Frankfurter currency, Free Dictionary, USGS earthquakes, HackerNews, Datamuse). Every record is derived from a real source, then JEV-deterministically composed into llms.txt datasets. Records stream into IndexedDB as they generate, cache in openwiki-v1, and package into a single .zip on export. No account, no telemetry, no server.
Search fans out to Wikipedia's action=query&list=search (paginated via continue), action=opensearch, then batches REST summary fetches for every page. Results dedupe, JEV-score, and — when the embedder is on — re-rank by a hybrid JEV + MiniLM score. Cached in IndexedDB under openwiki-v1. Expand fetches prop=extracts, prop=categories, prop=links; cached the same way. Nothing is fetched twice.
JEV is a 512-dimension float vector computed deterministically from text: unigrams with position weighting, bigrams, trigrams, and character 3-grams of longer tokens — all hashed into a shared space, weighted, and L2-normalized. Two JEV vectors compare with a single dot product, so ranking thousands of cached records against a query is instant with no model download. MiniLM, when on, fuses with JEV via a hybrid score 0.35·JEV + 0.65·MiniLM — never as a replacement.
Results stream in pages of 20 as you scroll. Loading is strictly scroll-triggered — the next page loads when a sentinel at the bottom of the results enters the viewport (400px margin). No auto-fill, no auto-load. A soft guard (default 1000 loaded results) keeps the DOM responsive; raise it with /limit <n> (or /limit 0 for unlimited). Use /loadall to pull the whole result set, then /export results json|csv|tsv|html|txt|md to siphon everything out. /export all json dumps current results and saved articles together.
| Kind | Shape |
|---|---|
qa | Multiple Q&A pairs extracted from the source |
instruction | ### Instruction / ### Response pairs |
definition | Term / definition glossary rows |
essay | Multi-paragraph composition from source sentences |
comparison | Table comparing the source concept to alternatives |
| Command | What it does |
|---|---|
/search <q> | Run the OpenWiki pipeline on a query — results panel with infinite scroll |
/auto <topic> | Start the auto research + data generation loop |
/gen <topic> | Generate one batch (single round) |
/expand <title> | Expand a Wikipedia article (extracts + categories + links) |
/limit <n> | Soft guard on loaded results (0 = unlimited) |
/loadall | Pull the whole result set at once |
/pause · /resume · /stop | Loop control |
/export | Package the dataset into a .zip with llms.txt master index |
/export results json|csv|tsv|html|txt|md | Export the current results panel |
/export all json | Dump current results + saved articles |
/schema | Edit the schema (kinds, sources, variations, volume, seed) |
/seed <n> | Set the deterministic seed |
/volume <n> | Set the target record count |
/kinds qa,instruction,... | Set record kinds |
/sources wiki,arxiv,... | Set API wings |
/limits | Show per-endpoint soft limiter status |
/endpoints | Show endpoint health |
/rerank on|off|status | MiniLM hybrid reranker control |
/model load tiny|small|large | Optional local LLM |
/stats | Current stats and dataset state |
/forget | Clear the dataset (records only) |
/clear | Clear the console display |
/speed slow|normal|fast | Typing rate |
/help | This dialog |
Every endpoint is protected by a token-bucket rate limiter, a global concurrency cap, an exponential-backoff circuit breaker, and a per-request deadline. A throttled source never blocks the answer — other wings keep producing. When all endpoints are cooling down, DataHawk pauses and resumes automatically once a token is available.
Records live in IndexedDB under datahawk-v1 / records, the wiki cache under openwiki-v1 / articles. The storage pill shows usage against the browser quota. Use /export to back up, then /forget to free space. DataHawk requests persistent storage the first time you generate a dataset.
A free tool from webxos.netlify.app. No account, no telemetry, no server. Query text leaves the browser only to the public APIs listed above. Records, schemas, seeds, and attachments stay in this browser. Export is a local download.
Deterministic: the same seed and topic always produce the same records.
Changes apply to the next /gen or /auto run. Existing records are not affected.