history4feed
Before you begin...
We use history4feed in the web version of Obstracts which includes many additional features over those in this codebase. You can find out more about the web version here.
Overview
It is common for feeds (RSS or XML) to only include a limited number of posts. I generally see the latest 3 - 5 posts of a blog in a feed. For blogs that have been operating for years, this means potentially thousands of posts are missed.
There is no way to page through historic articles using an RSS or ATOM feed (they were not designed for this), which means the first poll of the feed will only contain the limited number of articles in the feed. This limit is defined by the blog owner.
history4feed can be used to create a complete history for a blog and output it as an RSS feed.
history4feed offers an API interface that;
- takes an RSS / ATOM feed URL
- downloads a Wayback Machine archive for the feed
- identified all unique blog posts in the historic feeds downloaded
- downloads a HTML version of the article content on each page
- stores the post record in the databases
- exposes the posts as JSON or XML RSS
Install
Download and configure
# clone the latest code
git clone https://github.com/muchdogesec/history4feed
Configuration options
history4feed has various settings that are defined in an .env file.
To create a template for the file:
cp .env.example .env
To see more information about how to set the variables, and what they do, read the .env.markdown file.
Build the Docker Image
sudo docker compose build
Start the server
sudo docker compose up
Access the server
The webserver (Django) should now be running on: http://127.0.0.1:8002/
You can access the Swagger UI for the API in a browser at: http://127.0.0.1:8002/api/schema/swagger-ui/
Useful supporting tools
- Full Text, Full Archive RSS Feeds for any Blog
- An up-to-date list of threat intel blogs that post cyber threat intelligence research
- Donate to the Wayback Machine
Support
Minimal support provided via the DOGESEC community.
License
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file history4feed-1.2.12.tar.gz.
File metadata
- Download URL: history4feed-1.2.12.tar.gz
- Upload date:
- Size: 1.4 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c3f0192e9439ac7ffcc4b21c10b0ec609fe5dd86fd477cfe32ef2c227dd0ad9e
|
|
| MD5 |
f25f3586e2a3c6e80a5d0438a728bed0
|
|
| BLAKE2b-256 |
184c8dafbbd4d1d911687a980800b4f9429463ca434feaec2800694607c926cb
|
Provenance
The following attestation bundles were made for history4feed-1.2.12.tar.gz:
Publisher:
create-release.yml on muchdogesec/history4feed
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
history4feed-1.2.12.tar.gz -
Subject digest:
c3f0192e9439ac7ffcc4b21c10b0ec609fe5dd86fd477cfe32ef2c227dd0ad9e - Sigstore transparency entry: 2222779633
- Sigstore integration time:
-
Permalink:
muchdogesec/history4feed@faccde135929f4246e64298aa6611bcb15aca683 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/muchdogesec
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
create-release.yml@faccde135929f4246e64298aa6611bcb15aca683 -
Trigger Event:
push
-
Statement type:
File details
Details for the file history4feed-1.2.12-py3-none-any.whl.
File metadata
- Download URL: history4feed-1.2.12-py3-none-any.whl
- Upload date:
- Size: 52.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a674dcbfd64d3249f034e8450d12e3f2dbc2e818676a9ec1e09837a88685aff3
|
|
| MD5 |
e332ee80258ac24d7321ed6486fcc7be
|
|
| BLAKE2b-256 |
e33a84930650bfd8e85e5ff41f6438bdb2d340349584efedc69d11e00cc470bc
|
Provenance
The following attestation bundles were made for history4feed-1.2.12-py3-none-any.whl:
Publisher:
create-release.yml on muchdogesec/history4feed
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
history4feed-1.2.12-py3-none-any.whl -
Subject digest:
a674dcbfd64d3249f034e8450d12e3f2dbc2e818676a9ec1e09837a88685aff3 - Sigstore transparency entry: 2222780277
- Sigstore integration time:
-
Permalink:
muchdogesec/history4feed@faccde135929f4246e64298aa6611bcb15aca683 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/muchdogesec
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
create-release.yml@faccde135929f4246e64298aa6611bcb15aca683 -
Trigger Event:
push
-
Statement type:
