HTML landing page generator for LSST PDF documentation deployed from Git to LSST the Docs.
HTML landing page generator for LSST PDF documentation deployed from Git to LSST the Docs.
Lander works with Python 3.5 or above. You can install it from PyPI:
pip install lander
Run lander -h for command line help.
To create a landing page website, run lander with the local PDF file’s path:
lander --pdf <path>
The built PDF landing page site is available, by default, from the _build directory. View the site in a browser by running a Python web server:
cd _build && python -m http.server 8000 --bind 127.0.0.1
Get metadata from an lsstdoc document
With the --lsstdoc <tex path> argument, Lander will attempt to scrape metadata from the source of a lsstdoc-class LaTeX file, including:
- document handle
See https://lsst-texmf.lsst.io for information about the lsstdoc class.
Get metadata from the Travis environment
If you’re running on Travis CI, set the --env=travis to get metadata from Travis’s environment variables:
Lander tries to get as much metadata from the environment as possible (including from the Git repository and the LaTeX document itself), but sometimes this isn’t possible. In this case you can explicitly set metadata with these flags (see lander -h for more information):
- --authors (see note)
- --handle (such as LDM-151)
- --repo-url (such as https://github.com/lsst/ldm-151)
- --repo-branch (such as master)
- --date (such as 2017-05-22)
- --docushare-url (prefer the multi-version form, https://ls.st/ldm-151*)
--authors should be a JSON-formatted array, even for a single author. For example:
--authors "[\"First Author\", \"Second Author\"]"
Distributing extra files from the landing page
To include ancillary files with the main PDF document, provide their file paths with the --extra-downloads argument. These extra files are listed in the Downloads section of the landing page. The main PDF is always included first in this list.
Uploading to LSST the Docs
Lander works well with LSST the Docs. Lander can upload pages directly to LSST the Docs for you with these configurations:
- --upload — provide this flag to indicate a build should be uploaded.
- --ltd-product — Name of the product on LSST the Docs.
- --keeper-url or $LTD_KEEPER_URL.
- --keeper-user or $LTD_KEEPER_USER.
- --keeper-password or $LTD_KEEPER_PASSWORD.
- --aws-id or $LTD_AWS_ID.
- --aws-secret or $LTD_AWS_SECRET.
Note: these are advanced configurations and are typically added to a CI configuration automatically or by a Documentation Engineer. Reach out to #dm-docs on Slack for help.
You need both Python 3.5+ and node.js to develop Lander.
Initial set up
Clone and install dependencies (use a Python virtual environment of your choice):
git clone https://github.com/lsst-sqre/lander cd lander npm install -g gulp-cli npm install gulp assets pip install -e ".[dev]"
Run Python tests and linting
We use pytest for unit testing and style checks:
You can also run end-to-end trials of landing page builds:
These integration tests clone real LSST documents, compiles them with Docker, and builds landing pages to simulate continuous delivery workflows in production. Look for sites in _tests.
Build a development site
The default gulp workflow creates website assets and generates a test website:
Only build assets
gulp assets --env=deploy
This is how assets are built on CI for releases of Lander.
Developing CSS/Sass with squared
To make it easier to write Sass in squared while developing landing pages in Lander, we recommend linking a clone of squared to Lander’s node_modules. Assuming you’re starting from the lander/ root directory:
git clone https://github.com/lsst-sqre/squared ../squared npm link ../squared
- If you’re working on a branch in squared, then update squared’s version in package.json to that branch. For example: "squared": "lsst-sqre/squared#tickets/DM-10503". This allows Travis to install the development version of squared when testing Lander. Remember to make a release of squared before releasing a new version of Lander, see below.
- scss/app.scss in the lander repo imports Sass partials from squared and other packages (including inuitcss).
- If squared was modified, create a squared release first.
- Update package.json with the released version of squared. Using tagged npm releases is preferred to GitHub branches to make builds of releases repeatable.
- Create a signed tag: git tag -s 0.1.0 -m "v0.1.0". Use the PEP 440 schema.
- Push the tag: git push --tags. This will automatically create a Lander release on PyPI.
- Merge the development branch as necessary.
This project is open sourced under the MIT license. See LICENSE for details.
- Update Jinja to >=2.10.1 to address CVE-2019-10906.
- Update asset build pipeline to Gulp 4 and update all other npm dependencies at the same time.
- Update to lsst-projectmeta-kit to 0.3.5 for better author parsing.
- Update requests to 2.20.0 (security).
- Lander will now cleanly abort when building on Travis CI, but the secure environment variables are not available. This happens if the build is triggered by a fork.
- Switched to use $TRAVIS_BUILD_WEB_URL to get a URL to the build. This is better than computing the build URL during the travis-ci.org to travis-ci.com transition.
- Updated test dependencies to pytest 4.2.1.
- Switched to setuptools_scm for version string management.
- Update to lsst-projectmeta-kit 0.3.3 for improved detection of \input and \include commands in TeX source.
- Update to lsst-projectmeta-kit 0.3.1 for more reliable metadata.jsonld generation (works around pandoc issues converting some documents to plain text). There’s a new integration test make dmtn070 that demos this.
- Improve the testing strategy: - Run make pytest to run pytest with the correct arguments instead of using --add-opts in setup.cfg. This lets us run pytest directly with ad hoc arguments. - Run make test to run both pytest and the integration tests.
- Lander now creates and uploads a metadata.jsonld document alongside the landing page content (e.g., index.html). This content can be ingested by other tools into the LSST Projectmeta database.
- Switch from metasrc to lsst-projectmeta-kit 0.3.0. It’s the same package, but the new and rebranded lsst-projectmeta-kit includes the ability to generate JSON-LD from an LsstLatexDoc object.
- Change known domain for DocuShare from docushare.lsstcorp.org to docushare.lsst.org. This ensures that ls.st short links can continue to be verified.
- Create default values for abstract_plain and title_plain. This fixes cases where the abstract is not set in the underlying LaTeX document.
- Update metasrc to 0.2.2 to resolve issues with auto-downloading Pandoc in Travis CI (DM-12569).
- Update pytest to 3.2.5 and pytest-flake8 to 0.9.1 to fix incompatibilities in the floating indirect dependencies.
- Update metasrc to 0.2.1
- Use metasrc’s LsstLatexDoc.revision_datetime to obtain the date of a document. This method uses a combination of parsing the \date LaTeX command, looking at content Git commits, and falling back to ‘now’ to get an appropriate timestamp.
- Update metasrc to 0.2.0. This provides Pandoc integration for improved HTML rendering of content extracted from LaTeX documents.
- Improve how loggers are configured (warning level for third-party loggers, and info/debug levels for first-party LSST SQuaRE code).
- Update metasrc to 0.1.4. This update provides improved LaTeX command metadata extraction. (DM-11821)
- Temporarily skip ls.st and DocuShare-related unit tests because ls.st links to DocuShare are broken due to the DocuShare upgrade.
- Pin to metasrc 0.1.3
- Via metasrc, Lander has improved LaTeX source processing, including handling of referenced source files (\input and \include) and macros (\def and \newcommand).
- Improved treatment of draft status. The heuristic is that a document is considered a draft if the branch is not master and lsstdraft is not present in a lsstdoc document’s options.
- Fix logic for determining it Lander is running in a Travis PR environment.
- Log the Lander version at startup.
- Fixed Travis deployment issue. Used skip_cleanup: true to .travis.yml to prevent CSS and JS assets from bring cleaned up before creating a release.
- Detect if running from a Travis PR build (using the TRAVIS_PULL_REQUEST environment variable) and if so, abort the page build and upload. This is to prevent duplicate uploads from both branch and PR-based Travis jobs.
- Pin inuitcss to 6.0.0-beta4 because of the removal of rem functions in beta5.
- Update to metasrc>=0.1.1,<0.2.
- Use remove_comments and remove_trailing_whitespace feature from metasrc. This improves the accuracy of metadata extraction from tex source. For example, comment characters won’t appear in extract abstract content.
- Native PDF display via PDFObject.
- Multi-level metadata model for populating the landing page. Lander uses https://github.com/lsst-sqre/metasrc to extract content from the tex source (more work on this is needed). Otherwise, it gets data from environment variables (including Travis CI variables). Finally, metadata can be specified explicitly with command line arguments.
- Uses https://github.com/lsst-sqre/squared for CSS, icons, and logos. A Gulp and webpack workflow build these sources into deployable dependencies. Gulp and webpack are only needed by developers and CI; the deployable artifacts are included in PyPI releases.
- Release workflow is fully automated in Travis. Create a PEP 440 version tag, push it, and the release appears in PyPI.
- Includes an LSST the Docs upload client (via https://github.com/lsst-sqre/ltd-conveyor) built-in so that https://github.com/lsst-sqre/ltd-mason isn’t required.
- Usage and development docs are currently in the README.
Copyright (c) 2017 Association of Universities for Research in Astronomy, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Release history Release notifications | RSS feed
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size lander-0.2.0-py3-none-any.whl (38.7 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size lander-0.2.0.tar.gz (45.1 kB)||File type Source||Python version None||Upload date||Hashes View|