Ahrefs Top Websites Monthly Catalog Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Export the July 2026 Travel top 100 with the existing SimilarWeb workflow, then build a reusable monthly cross-category merge and local keyword browser for Games, Travel, and future categories.

Architecture: A generic sequential exporter reads category 01_domains.csv files and writes resumable per-domain JSON plus aggregate CSV. A separate catalog builder discovers category folders, adds category provenance, writes monthly merged CSVs, and generates file-protocol-safe index.html plus index_data.js for every category and the monthly catalog.

Tech Stack: Python 3 standard library, local Express/Puppeteer SimilarWeb API, Node test suite, dependency-free HTML/CSS/JavaScript, Chromium browser QA.

---

Task 1: Travel source list and generic exporter

Files: - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/01_domains.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/test_export_category.py - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/export_category.py

• [x] Write failing tests that load a 100-row source CSV, reject duplicate/non-contiguous ranks, flatten detailed keyword rows, and flatten landing-page rows.
• [x] Run python3 -m unittest -v test_export_category.py and confirm failure because export_category.py does not exist.
• [x] Implement load_domains(), flatten_keyword_payload(), flatten_landing_payload(), cache validation, sequential retrying fetches, aggregate CSV generation, and summary generation.
• [x] Run the tests and confirm all pass.

Task 2: Travel full export

Files: - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/raw/site_keywords/*.json - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/raw/landing_pages/*.json - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/02_site_keywords_all.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/03_landing_pages_all.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/errors.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/summary.json

• [x] Refresh the SimilarWeb session only if a real data request returns HTTP 401, then start the local API.
• [x] Run one-domain validation and require non-empty detailed keyword and landing-page rows.
• [x] Run the exporter for all 100 domains with cache reuse and retries.
• [x] Re-run any failures and rebuild the category aggregates from cache.

Task 3: Cross-category merge and index generator

Files: - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/test_build_catalog.py - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/build_catalog.py - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/index_template.html

• [x] Write failing tests for category discovery, provenance columns, merged row counts, JavaScript-safe serialization, and the three exact keyword action URL builders.
• [x] Run python3 -m unittest -v test_build_catalog.py and confirm failure because the catalog builder does not exist.
• [x] Implement discovery and CSV merging for ahrefs-top-*-worldwide-2026-07 directories while excluding the monthly catalog directory.
• [x] Implement index_data.js generation and a dependency-free template with search, category/domain filters, pagination, top-URL links, and Google/Query.Domains/Trends buttons.
• [x] Run all Python tests and confirm they pass.

Task 4: Generate and browser-test all indexes

Files: - Create: /Users/project/1B/sites/关键词/ahrefs-top-games-worldwide-2026-07/index.html - Create: /Users/project/1B/sites/关键词/ahrefs-top-games-worldwide-2026-07/index_data.js - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/index.html - Create: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/index_data.js - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/01_all_categories_domains.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/02_all_categories_site_keywords.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/03_all_categories_landing_pages.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/04_category_summary.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/index.html - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/index_data.js

• [x] Build all category and monthly outputs and assert merged row counts equal category row-count sums.
• [x] Open the Games, Travel, and monthly index.html files in Chromium and verify initial render, filtering, pagination, top URL, and all three encoded keyword actions.
• [x] Run the complete Node and Python test suites plus a fresh JSON/CSV integrity audit.
• [x] Stop the dedicated SimilarWeb API and browser after all verification succeeds.

Task 5: Sort every visible keyword table column

Files: - Modify: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/test_build_catalog.py - Modify: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/index_template.html - Regenerate: /Users/project/1B/sites/关键词/ahrefs-top-games-worldwide-2026-07/index.html - Regenerate: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/index.html - Regenerate: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/index.html

• [x] Add a failing template test requiring 15 sortable headers, sort state, numeric metric parsing, direction indicators, and filter-before-sort-before-pagination flow.
• [x] Run python3 -m unittest -v test_build_catalog.py and confirm failure because sortable header markup is absent.
• [x] Replace static header text with accessible sort buttons and implement stable single-column sorting with numeric/text comparators and null-last handling.
• [x] Rebuild the three indexes and run all Python tests.
• [x] Browser-test category, keyword, abbreviated numeric, signed-change, and empty-value sorting on the monthly index, then check console errors.

Task 6: Remove pagination and fix the table workspace in place

Files: - Modify: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/test_build_catalog.py - Modify: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/index_template.html - Regenerate: /Users/project/1B/sites/关键词/ahrefs-top-games-worldwide-2026-07/index.html - Regenerate: /Users/project/1B/sites/关键词/ahrefs-top-travel-worldwide-2026-07/index.html - Regenerate: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/index.html

• [x] Add failing template tests that reject pagination markup/state, require every filtered-and-sorted row to render, and require a fixed viewport/table-header layout.
• [x] Run python3 -m unittest -v test_build_catalog.py and confirm the new tests fail against the paginated template.
• [x] Remove page-size and previous/next controls, render every sorted row through one document fragment, and make only the table data region scrollable.
• [x] Rebuild the three indexes and run all Python tests.
• [x] Browser-test all-row counts, filtering, full-data sorting, fixed header positions, horizontal scrolling, responsiveness, and console errors.

Task 7: Generate a lightweight positive traffic-growth index

Files: - Modify: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/test_build_catalog.py - Modify: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/build_catalog.py - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/growth_template.html - Generate: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/05_positive_traffic_growth.html

• [x] Add failing tests for parsing positive percentages, rejecting non-positive/special values, descending numeric order, one-file output, and 200-row progressive rendering.
• [x] Run python3 -m unittest -v test_build_catalog.py and confirm failure because the growth-index functions do not exist.
• [x] Implement positive-growth selection, stable numeric sorting, standalone HTML generation, and progressive scroll rendering.
• [x] Rebuild the monthly catalog and run all Python and SimilarWeb extraction tests.
• [x] Browser-test embedded row count, default descending order, initial/next batch sizes, filters, fixed header, load timing, and console errors.

Task 8: Add Sports, Shopping, and Science Top 100 sources

Files: - Modify: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/test_export_category.py - Create: /Users/project/1B/sites/关键词/ahrefs-top-sports-worldwide-2026-07/01_domains.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-shopping-worldwide-2026-07/01_domains.csv - Create: /Users/project/1B/sites/关键词/ahrefs-top-science-worldwide-2026-07/01_domains.csv

• [x] Add failing source-contract tests requiring 100 unique domains, ranks 1–100, the expected category, and July 2026 metadata for all three lists.
• [x] Run python3 -m unittest -v test_export_category.py and confirm failure because the three source files do not exist.
• [x] Transcribe all 300 Ahrefs rows with rank, search traffic, traffic change, and rank change.
• [x] Run source tests and an independent duplicate/rank audit.

Task 9: Export the three new SimilarWeb categories

Files: - Generate under each new category: raw/site_keywords/*.json, raw/landing_pages/*.json, 02_site_keywords_all.csv, 03_landing_pages_all.csv, errors.csv, summary.json, index.html, index_data.js

• [x] Start the local SimilarWeb wrapper and validate one uncached Sports domain through both endpoints.
• [x] Export Sports, Shopping, and Science with per-domain cache writes and retries.
• [x] Re-run failures until every domain has both caches or a documented terminal error.
• [x] Rebuild each aggregate and verify CSV row counts equal cached JSON row sums.

Status on 2026-07-14: Sports and Shopping are complete at 100/100 caches for both endpoints. Science is safely paused at 45 keyword caches and 46 landing-page caches because the provider returned the account-level daily-quota page; the wrapper now reports that page as HTTP 429 and the catalog builder excludes this incomplete category until resume succeeds.

Status on 2026-07-15: Science resumed and completed at 100/100 for both endpoints. Reference also completed at 100/100 after fixing shared-page request serialization and stale hash-route validation in the local SimilarWeb wrapper. Real Estate and Pets & Animals use the same resumable export contract.

Completed on 2026-07-16: Real Estate, Pets & Animals, and People & Society all have 100/100 valid caches for both endpoints. The nine-category catalog now contains 900 ranked domains, 89,473 keyword rows, and 88,592 landing-page rows. bk33.org is the single explicitly confirmed empty keyword response after repeated clean-session verification; its audit metadata is retained instead of fabricating rows.

Task 10: Build the >500% three-word query.domains report

Files: - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/test_growth_domain_report.py - Create: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/build_growth_domain_report.py - Generate: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/06_growth_keywords_over_500_3_words.csv - Generate: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/07_query_domains_keyword_domains.csv - Generate: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/08_growth_keywords_query_domains.html - Cache: /Users/project/1B/sites/关键词/ahrefs-top-websites-worldwide-2026-07/raw/query_domains/*.json

• [x] Add failing tests for strict percentage filtering, whitespace word count, stable numeric sorting, Unicode/IDNA candidate generation, response parsing, traffic summaries, credential-free caches, and standalone report behavior.
• [x] Run the focused tests and confirm failure because the report module does not exist.
• [x] Implement the resumable API client using runtime QUERY_DOMAINS_API_KEY, batched status checks, registered-only WHOIS/traffic calls, retries, and immediate cache writes.
• [x] Rebuild the catalog, filter and deduplicate keywords, query every valid candidate, then generate CSV and standalone HTML reports.
• [x] Run the final source, exporter, catalog, report, and Node tests; audit cache/CSV totals and scan all generated files for credential leakage.
• [x] Open the final standalone reports in Chromium; verify counts, filters, numeric sorting, query links, progressive rendering, and console errors.

Task 11: Add Reference, Real Estate, Pets & Animals and the strict available .com report

Files: - Create the three category 01_domains.csv source lists and resumable SimilarWeb caches. - Create build_available_com_report.py, available_com_template.html, and test_available_com_report.py. - Generate 09_available_com_growth_over_1000_traffic_over_300k.csv and 10_available_com_growth_over_1000_traffic_over_300k.html.

• [x] Transcribe and validate 100 unique, contiguous Ahrefs domains for all three categories.
• [x] Finish Science and Reference at 100/100 keyword and landing-page caches.
• [x] Add a query.domains check-only path that never requests WHOIS or traffic.
• [x] Implement strict >1000%, >300K, 1–3 word filtering, representative-row aggregation, .com normalization, available-only output, and a standalone sortable HTML.
• [x] Complete Real Estate and Pets & Animals at 100/100 for both endpoint caches.
• [x] Add People & Society and complete all 100 domains for both endpoint caches.
• [x] Rebuild the nine-category catalog, query the final unique .com candidates, and generate 09/10 with 377 confirmed available domains.
• [x] Run cross-file totals, browser behavior, and exact-secret leakage verification.

Task 12: Build the lower-decay monetization shortlist

Files: - Create build_monetizable_opportunities.py, monetizable_opportunities_template.html, and test_monetizable_opportunities.py. - Generate 11_monetizable_fresh_opportunities.csv and 12_monetizable_fresh_opportunities.html.

• [x] Add tests for short-event, person, brand/media and anomalous-site exclusion; commercial prioritization; medical/seasonal caution; current click momentum; and standalone sortable HTML behavior.
• [x] Score the strict available candidates using traffic, growth, click momentum, KD, CPC/intent and durability.
• [x] Generate 3 priority rows and 18 watch rows from the 377 strict available .com candidates.
• [x] Complete final browser, cross-file and secret-leakage verification.

Task 13: Add Online Communities, News and Law & Government

Files: - Create the three category 01_domains.csv lists and full resumable SimilarWeb exports. - Regenerate the monthly catalog and every dependent report.

• [x] Transcribe and test 300 Ahrefs rows with contiguous ranks, unique domains and July 2026 metadata.
• [x] Export site-keyword and landing-page endpoints for every source domain.
• [x] Independently retry Facebook, Reddit and TikTok in clean sessions, then retain their stable zero responses as explicitly confirmed empty audit records.
• [x] Rebuild the 12-category catalog with 1,200 ranked records, 119,156 keyword rows and 117,627 landing-page rows.
• [x] Add controlled concurrency to query.domains availability batches while preserving per-domain atomic caches and resumability.

Task 14: Build strict business niches and a separate AI/Web SaaS watchlist

Files: - Create build_high_intent_business_niches.py, high_intent_business_niches_template.html, and test_high_intent_business_niches.py. - Generate 13_high_intent_business_niches.csv and 14_high_intent_business_niches.html. - Create build_ai_web_saas_watchlist.py and test_ai_web_saas_watchlist.py. - Generate 15_ai_web_saas_watchlist.csv and 16_ai_web_saas_watchlist.html.

• [x] Enforce traffic >1M, KD <30, average volume 10K–99.9K, commercial/product intent and semantic exclusion in the strict report.
• [x] Add a regression test excluding mexico jersey as a place-name term; retain the honest zero-result strict report.
• [x] Build a separately labelled sub-million watchlist requiring positive growth, KD <30, average volume in the tens of thousands and a productizable AI/Web-SaaS family.
• [x] Mark platform downloader/viewer terms as high policy risk and low-CPC utilities as traffic tools rather than paid-SaaS proof.
• [x] Generate single-page, fixed-header, full-column-sortable HTML with Google, query.domains and Google Trends actions.
• [x] Refresh query.domains reports to 487 confirmed available .com candidates, then rebuild the lower-decay shortlist to 3 priority and 19 watch rows.
• [x] Query all 24 watchlist .com candidates: 22 available and 2 registered, with registered-date and domain-traffic fields retained where returned.

Task 15: Add Jobs & Education, Internet & Telecom, Home & Garden and Hobbies & Leisure

Files: - Create the four category 01_domains.csv lists and full resumable SimilarWeb exports. - Regenerate the monthly catalog and all dependent query.domains and opportunity reports.

• [x] Transcribe and test 400 Ahrefs rows with contiguous ranks, unique domains and July 2026 metadata.
• [x] Export both SimilarWeb endpoints for all 400 domains; independently retry transient empty responses instead of caching false zeroes.
• [x] Complete 100/100 endpoint coverage for every new category: 39,933 keyword rows and 38,987 landing-page rows.
• [x] Rebuild the 16-category catalog with 1,600 ranked records, 159,089 keyword rows and 156,614 landing-page rows.
• [x] Refresh 09/10 to 498 confirmed available .com candidates and 11/12 to 2 priority plus 20 watch rows.
• [x] Retain the honest zero-result strict business report and refresh the AI/Web SaaS watchlist to 25 keywords: 22 exact .com candidates available and 3 registered.

Task 16: Add Health, Food & Drink and Finance

Files: - Create and validate the three category 01_domains.csv files from the July 2026 Ahrefs Worldwide lists. - Export both SimilarWeb endpoints with resumable per-domain caches, then regenerate the catalog and downstream reports.

• [x] Transcribe and test all three Top 100 source lists with contiguous ranks, unique domains and complete Ahrefs metadata.
• [x] Complete Health at 100/100 endpoint coverage with 10,000 keyword rows and 9,917 landing-page rows.
• [x] Complete Food & Drink at 100/100 endpoint coverage with 10,000 keyword rows and 9,921 landing-page rows; independently retry and recover the transient toprecepty.cz empty response.
• [x] Preserve the Finance checkpoint after the provider reported account-level daily quota exhaustion: 25 keyword caches and 24 landing-page caches, totaling 2,500 and 2,400 rows.
• [x] Keep incomplete Finance out of the catalog and rebuild the 18-category total with 1,800 ranked records, 179,089 keyword rows and 176,452 landing-page rows.
• [x] Refresh 09/10 to 500 confirmed available .com candidates, retain 2 priority plus 20 watch rows in 11/12, and retain the honest zero-result strict business report.
• [x] Refresh the AI/Web SaaS watchlist to 25 keywords: 22 exact .com candidates available and 3 registered.
• [ ] After the SimilarWeb provider resets the daily quota, resume Finance to 100/100 endpoint coverage, then rebuild the 19-category catalog and all dependent reports.

Task 17: Make the 179K-row local catalog responsive

• [x] Store the monthly total index as compact array rows with one shared 15-field schema instead of repeating object keys and unused fields.
• [x] Keep filtering and sorting global while rendering only the first 200 DOM rows and appending further batches on table scroll or explicit load.
• [x] Preserve the fixed masthead, controls and table headers, all 15 sortable columns, local file:// loading and the three keyword actions.
• [x] Reduce index_data.js from about 89 MB to about 32 MB.
• [x] Verify in real headless Chrome: 179,089 logical rows, 200 initial rows, scroll growth, full-set sort, category/keyword filters, query.domains prefill and zero console errors.

Task 18: Restore quantitative filters on the monthly total index

• [x] Add 28-day traffic K/M presets plus free-form minimum and maximum inputs with K/M/B parsing.
• [x] Add positive, +100%, +500%, and +1000% growth presets plus signed percentage minimum and maximum inputs.
• [x] Keep every quantitative filter composable with keyword, domain and category filters, with a one-click reset and invalid-input state.
• [x] Verify in real headless Chrome against all 179,089 rows, including K/M ranges, traffic bounds, growth presets, growth bounds, combined filtering, reset, fixed table header and zero console errors.

本文档为站内渲染。原始文件本地路径：saas/source/keywords/关键词-ahrefs-top-worldwide-2026-07-all-ahrefs-top-websites-wor-a2f2ea.md（仅本地保留，不入库不部署）