markdown-analyzer-lib 0.1.0

A Python library for parsing, analyzing, and converting Markdown and MDX documents, and converting websites to Markdown.

Markdown Analyzer Library

A Python library for parsing, analyzing, and converting Markdown and MDX documents. It also includes capabilities to scrape websites and convert their content into structured Markdown.

Features

• Parse Markdown and MDX into an Abstract Syntax Tree (AST)-like structure.
• Identify various Markdown elements: headers, paragraphs, lists, code blocks, tables, links, footnotes, etc.
• Analyze document statistics: word count, character count, element counts.
• Convert HTML content to Markdown.
• Scrape websites and convert entire sites into a single Markdown document.
• Basic MDX parsing support.

Installation

pip install markdown-analyzer-lib

(Once published to PyPI)

To install locally for development:

# Navigate to the markdown_analyzer_lib_project directory
pip install .
# or for an editable install
pip install -e .

User Documentation

This section provides detailed information on how to use the Markdown Analyzer Library.

Running the Library

The Markdown Analyzer Library is designed to be used as a Python library integrated into your own Python scripts or applications.

As a Python Library

To use the library, you first need to import the relevant classes. The primary classes are MarkdownDocument for handling individual Markdown files or strings, and MarkdownSiteConverter for converting websites.

1. Analyzing a Local Markdown File:

You can parse and analyze a Markdown file directly from your file system.

from markdown_analyzer_lib import MarkdownDocument

# Create a MarkdownDocument object from a file
try:
    doc = MarkdownDocument.from_file("path/to/your/document.md")
except FileNotFoundError:
    print("Error: The specified file was not found.")
    exit()
except Exception as e:
    print(f"An error occurred: {e}")
    exit()

# Get a summary of the document
# Includes word count, character count, and counts of various elements
summary = doc.get_summary()
print("Document Summary:")
for key, value in summary.items():
    print(f"- {key.replace('_', ' ').capitalize()}: {value}")

# Get all headers
headers = doc.get_headers()
print("\nHeaders:")
if headers:
    for header in headers:
        print(f"- Level {header['level']}: {header['text']} (Line {header['line']})")
else:
    print("No headers found.")

# Get all paragraphs
paragraphs = doc.get_paragraphs()
print("\nParagraphs:")
if paragraphs:
    for i, para in enumerate(paragraphs):
        print(f"Para {i+1} (Line {para['line']}): {para['text'][:100]}...") # Print first 100 chars
else:
    print("No paragraphs found.")

# Get all links
links = doc.get_links()
print("\nLinks:")
if links:
    for link in links:
        print(f"- Text: {link['text']}, URL: {link['url']} (Line {link['line']})")
else:
    print("No links found.")

# Get all code blocks
code_blocks = doc.get_code_blocks()
print("\nCode Blocks:")
if code_blocks:
    for i, block in enumerate(code_blocks):
        lang = block.get('language', 'N/A')
        print(f"Block {i+1} (Language: {lang}, Line {block['line']}):")
        print(block['code'][:150] + "..." if len(block['code']) > 150 else block['code'])
else:
    print("No code blocks found.")

# Get raw AST-like structure (for advanced use)
# ast_structure = doc.get_ast()
# print("\nAST Structure (excerpt):")
# print(str(ast_structure)[:500] + "...")

2. Analyzing Markdown from a String:

If you have Markdown content as a string, you can parse it directly.

from markdown_analyzer_lib import MarkdownDocument

markdown_string = """
# My Document Title
This is the first paragraph. It contains some *italic* and **bold** text.

## Section 1
- List item 1
- List item 2

Another paragraph here.
"""
doc_from_string = MarkdownDocument.from_string(markdown_string)

print("Summary from String:")
summary = doc_from_string.get_summary()
for key, value in summary.items():
    print(f"- {key.replace('_', ' ').capitalize()}: {value}")

print("\nHeaders from String:")
headers = doc_from_string.get_headers()
for header in headers:
    print(f"- Level {header['level']}: {header['text']}")

3. Converting a Website to Markdown:

The library can scrape a website and convert its content into a single Markdown document. This is useful for archiving web content or analyzing website structure.

Note: Web scraping should be done responsibly and in compliance with the website's robots.txt and terms of service.

from markdown_analyzer_lib import MarkdownSiteConverter

# Initialize the converter with the base URL of the site you want to scrape
# max_depth controls how many levels of links to follow from the base URL
# output_file is where the combined Markdown content will be saved
# Be cautious with max_depth, as it can lead to very large outputs and long processing times.
# Start with max_depth=0 (only the base URL) or max_depth=1.

# Example (commented out to prevent accidental execution during tests):
# try:
#     site_converter = MarkdownSiteConverter(
#         base_url="https://example.com", # Replace with a real, simple site for testing
#         max_depth=0, # Only scrape the initial page
#         output_file="website_content.md"
#     )
#     print(f"Starting website conversion for {site_converter.base_url}...")
#     markdown_output_path = site_converter.convert_site_to_markdown()
#     if markdown_output_path:
#         print(f"Website converted and saved to: {markdown_output_path}")
#         # You can then analyze this generated Markdown file:
#         # site_doc = MarkdownDocument.from_file(markdown_output_path)
#         # print("\nSummary of scraped website content:")
#         # print(site_doc.get_summary())
#     else:
#         print("Website conversion failed or produced no output.")
# except Exception as e:
#     print(f"An error occurred during website conversion: {e}")

# For a practical example, ensure you have a test website or use a simple, permissive one.
# For instance, if you have a local server running at http://localhost:8000 with some HTML files:
# site_converter_local = MarkdownSiteConverter(
# base_url="http://localhost:8000",
# max_depth=1,
# output_file="local_site_content.md"
# )
# local_site_markdown = site_converter_local.convert_site_to_markdown()
# if local_site_markdown:
# print(f"Local site converted to {local_site_markdown}")

User Manual

This manual provides guidance on how to effectively use the library for common tasks.

Task 1: Getting Document Statistics Use the get_summary() method on a MarkdownDocument object.

doc = MarkdownDocument.from_file("my_document.md")
summary = doc.get_summary()
print(f"Word Count: {summary['word_count']}")
print(f"Paragraph Count: {summary['paragraph_count']}")

Task 2: Extracting All Links from a Document Use the get_links() method.

doc = MarkdownDocument.from_file("my_document.md")
links = doc.get_links()
for link in links:
    print(f"Link Text: {link['text']}, URL: {link['url']}")

Task 3: Finding Specific Headers Iterate through the results of get_headers() and filter by level or text.

doc = MarkdownDocument.from_file("my_document.md")
h2_headers = [h for h in doc.get_headers() if h['level'] == 2]
print("All H2 Headers:")
for h2 in h2_headers:
    print(f"- {h2['text']}")

Task 4: Converting an Online Article to Markdown for Offline Reading Use MarkdownSiteConverter with max_depth=0 pointing to the article's URL.

from markdown_analyzer_lib import MarkdownSiteConverter
# article_converter = MarkdownSiteConverter(
#     base_url="URL_OF_THE_ARTICLE_HERE",
#     max_depth=0,
#     output_file="article.md"
# )
# article_converter.convert_site_to_markdown()
# print("Article saved to article.md")

API Reference (To Be Expanded)

Detailed API documentation for all classes and methods will be available here or can be generated using tools like Sphinx. For now, please refer to the source code docstrings for detailed information on parameters and return values.

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

License

This project is licensed under the MIT License - see the LICENSE file for details.