Introducing the NeedSift Product Activity Scanner

Discovery needs evidence that a product is alive

A beautiful launch page can remain online for years after development stops. At the same time, a small team may ship meaningful improvements every week without creating a large launch campaign. A static directory cannot tell these situations apart.

NeedSift now builds a daily product history from public, low-cost signals. The goal is not surveillance. It is a compact answer to a useful question: is this product reachable, maintained, publishing, and shipping?

What the scanner observes

The worker checks the primary website, stable release metadata from App Store and Google Play, public GitHub or Telegram activity when available, and sitemap files advertised through robots.txt. Additional demo links can be checked through a normalized content fingerprint.

For every daily observation it stores only the information needed for history: availability, response time, content fingerprints, release version and date, newest public activity, sitemap URL counts, and the number of newly discovered pages.

• Website reachability and meaningful visible-content changes.
• App version and release date from public store metadata.
• Newest public GitHub or Telegram activity where supported.
• New sitemap URLs and recent last-modified dates.
• A daily score that can be plotted without exposing private analytics.

Sitemap tracking rewards durable publishing

When a product publishes documentation, changelog entries, guides, integrations, or landing pages, those additions often appear in its sitemap. NeedSift records up to 2,000 URLs and can detect pages that did not exist during the previous scan.

The scanner does not visit those pages. It reads a maximum of three same-domain sitemap documents, limits each response to two megabytes, and uses the first observation only as a baseline. This makes publishing visible without turning NeedSift into an aggressive crawler.

Polite by design

The worker runs once per day with limited concurrency and a delay between requests to the same hostname. Every request has a timeout, response bodies are capped, redirects are limited, and unsupported sources are skipped instead of retried repeatedly.

NeedSift never needs customer credentials, private dashboards, or analytics access. The scanner uses the same public information a visitor or search engine can see, but reads much less than a search crawler.

What builders will see next

Product pages now include an activity chart and separate signals for website health, releases, social publishing, and sitemap activity. As the history grows, builders will be able to understand which kinds of work create sustained momentum.

Future improvements will focus on stronger anomaly detection, clearer explanations for score changes, and better recognition of changelogs and documentation—without increasing crawl pressure.