comment-scraper-haystack

A Haystack converter for Reddit and Hacker News threads you exported with Comment Scraper. CommentScraperConverter turns each comment into one Haystack Document. Comments keep the numbers the extension's Markdown export gives them, so [c12] in a model's answer is comment 12 in your file.

The converter reads JSON files on your disk. It makes no network requests, Reddit and Hacker News included.

Install

pip install comment-scraper-haystack

You need Python 3.10 or newer and Haystack 2.9 or newer (haystack-ai). Haystack 3 works too.

Export a thread

1. Install the Comment Scraper extension for Chrome or Edge from comment-scraper.com. Exports are free and don't need an account.
2. Open a Reddit or Hacker News thread and open the extension's side panel.
3. Pick JSON, one row per comment or JSON, replies nested, then export and download the file. CSV and Markdown exports can't be converted.

Convert it

from haystack_integrations.components.converters.comment_scraper import CommentScraperConverter

converter = CommentScraperConverter()
result = converter.run(sources=["which-crm-do-small-teams-actually-stick-with.json"])

for document in result["documents"][1:4]:
    meta = document.meta
    print(meta["comment_number"], meta["reply_to_number"], document.content)

1 None We left our CRM after the renewal doubled.
2 1 Ask for the year-two price in writing.
3 2 The export button was the first thing they hid.

That output comes from tests/fixtures/which-crm.json, a made-up thread used in the tests. The first Document is the thread's post, so the comments start at index 1. Comment 2 replies to comment 1, which the Markdown export shows as [c2] reply to [c1].

sources takes file paths or ByteStream objects. A batch export holds several threads in one file, and you get Documents for all of them, thread by thread.

run() also takes meta, like Haystack's own converters: one dict for every Document, or a list with one dict per source. The keys are added to each Document's meta. If one of your keys has the same name as a comment field below, the comment's value wins.

Three options when you create the converter:

• include_post=True (the default) gives the thread's post first, as its own Document. Its content is the title, a blank line, then the post body.
• include_deleted=False (the default) skips deleted and removed comments, because they have no text. They keep their numbers, so the numbering still matches the Markdown export. Set it to True to get them with (deleted) or (removed) as their content.
• store_full_path=False (the default) puts only the file name in meta["file_path"]. Set it to True for the full path.

Write the comments to a document store

from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.components.converters.comment_scraper import CommentScraperConverter

store = InMemoryDocumentStore()

indexing = Pipeline()
indexing.add_component("converter", CommentScraperConverter())
indexing.add_component(
    "writer", DocumentWriter(document_store=store, policy=DuplicatePolicy.OVERWRITE)
)
indexing.connect("converter.documents", "writer.documents")
indexing.run({"converter": {"sources": ["which-crm-do-small-teams-actually-stick-with.json"]}})
print(store.count_documents())

retriever = InMemoryBM25Retriever(document_store=store, top_k=3)
only_comments = {"field": "meta.record_type", "operator": "==", "value": "comment"}
for document in retriever.run(query="renewal price", filters=only_comments)["documents"]:
    print(f"[c{document.meta['comment_number']}] {document.content}")

7
[c2] Ask for the year-two price in writing.
[c1] We left our CRM after the renewal doubled.
[c8] Same here, the renewal email came with no warning.

Seven Documents: the post and the six comments that have text. To use another document store, pass it to DocumentWriter and check what meta values it accepts (see the note on None below). For embedding retrieval, put an embedder between the converter and the writer.

Each Document's id is site:thread_id:comment_id, for example reddit:crm1:a3, and site:thread_id for the post. Export the thread again later, write it with DuplicatePolicy.OVERWRITE, and its comments are replaced, not added twice. With the default options, a comment deleted since your last import, or one the new export doesn't include, keeps its old Document. To refresh a thread completely, convert the new export first. If it gives Documents for the thread, delete the thread's old ones, filtering on meta.site and meta.thread_id, and write the new ones. Check the fetch counts at the top of the new file before you do: comments a partial export lacks are gone afterwards.

What each Document holds

content is the comment text as the export has it: Reddit's own Markdown when the extension read Reddit's JSON, and plain text for Hacker News or when it had to read the Reddit page instead.

Meta key	What it holds
file_path	Name of the export file (the full path with store_full_path=True); missing for a ByteStream without one
site	reddit or hn
thread_id, thread_title	The thread's ID and title
thread_url	Link to the thread
subreddit	Subreddit name; None for Hacker News
fetched_at	When the extension fetched the thread, ISO 8601 in UTC
record_type	post or comment
comment_id	The site's comment ID, as text; None for the post
comment_number	N in [cN], the number the Markdown export shows; None for the post. Numbering starts again in each thread of a batch export
parent_id	ID of the comment it replies to; the thread ID for top-level comments
reply_to_number	Number of the comment it replies to; None for top-level comments and when the parent wasn't fetched
parent_missing	True when the parent comment wasn't fetched
depth	0 for top-level comments, 1 for their replies, and so on
site_score	Score as shown on the site; None when hidden. Hacker News doesn't show comment scores
score_hidden	True when Reddit hides the score
created_utc, edited_utc	When it was posted and last edited, ISO 8601 in UTC; edited_utc is None if never edited
is_op	True when the person who started the thread wrote it
comment_state	normal, deleted or removed
permalink	Link to the comment

Exports don't contain usernames, so there's no author field. Names someone typed inside a comment are kept as written.

The score is site_score, not score, because score is the Document field retrievers fill with relevance. Haystack 2.x and 3.0 put meta keys and Document fields in one flat dict when a document store saves or returns a Document, so a meta key called score would be lost.

All values are strings, numbers, booleans or None, and nothing is nested, so you can filter on any of them, as meta.record_type above. If your document store rejects None values, drop those keys before you write the Documents.

When a source can't be converted

Like Haystack's own converters, the converter logs a warning and skips the source, then carries on with the rest. The warning names the source and says why:

• the file can't be read
• it isn't a Comment Scraper JSON export (a CSV or Markdown export, say)
• a thread in it is damaged, for example cut off part way, or with a field removed or changed to the wrong type by hand; the warning says which thread and which field
• a nested export's replies go deeper than Python's JSON parser allows; export the thread as "JSON, one row per comment" instead

An export can be partial: Reddit may have rate-limited the extension, or you stopped it early. The file header records how many comments were fetched and how many the site reports. The converter passes on what's in the file and doesn't fill gaps.

Privacy

The converter reads the files and ByteStreams you pass to it and nothing else. It doesn't write files and doesn't connect to the internet; the tests run with network access disabled to check that.

Haystack itself sends anonymous usage telemetry when a pipeline runs. Set HAYSTACK_TELEMETRY_ENABLED=False before importing Haystack to turn it off. If you send the comments to a hosted model or a hosted document store, they go to that provider under its terms.

Development

uv sync
make test   # tests, with network access disabled
make lint   # ruff and mypy

The tests include indexing and retrieval pipelines, and they run the Python examples in this README and check their output.

The fixtures in tests/fixtures are made-up threads written by the extension's own export code, together with the [cN] line the Markdown export prints for each comment. make fixtures rebuilds them; it needs a checkout of comment-scraper-mcp next to this repository, with its dependencies installed.

To check real exports, which aren't committed, run CS_REAL_EXPORTS_DIR=/path/to/exports uv run pytest tests/test_real_exports.py. If a thread's .md export sits next to its .json, the comment numbers are compared line by line.

Support

Open an issue on GitHub or email hello@comment-scraper.com.

License

MIT. Comment Scraper isn't affiliated with Reddit, Inc. or Y Combinator.