snovault 1.0.67

Snovault Hybrid Object Relational Database Framework

SnoVault JSON-LD Database Framework

System Installation (OSX Big Sur(testing), Catlina(10.15.x), Mojave(10.14.6))

We will try to keep this updated as OSX, Xcode, and brew update. However the steps below are examples and not guaranteed to work for your specific system. See the dependency’s website for detailed instructions or let us know of any changes with a pull request.

1. Command line tools

   xcode-select --install

2. brew: https://brew.sh/ . Make sure git is installed

3. Python 3.8.5

4. Postgres@11 (Postgres@9.3 also works)

   brew install postgresql@11
   # May need to add postgres to PATH in your shell profile, e.g. ~/.bash_profile, ~/.zshrc
   # echo 'export PATH="/usr/local/opt/postgresql@11/bin:$PATH"' >> YOUR_SHELL_PROFILE

5. Node 10.x.x

   brew install node@12

   You may need to link node/npm with brew link node@12 then add it to your PATH

6. Ruby - Non system version to install compass while avoiding permission errors

   brew install ruby
   # May need to add ruby to your bash_profile/zshrc and restart terminal

7. Compass

   gem install compass
   # Test the install
   compass -v
   # If the command is not found then find your ruby bin directory
   ls /usr/local/lib/ruby/gem/
   # If you have two versions use the active one
   ruby -v
   # Using the correct ruby version bin diretory, make a sym link
   ln -s /usr/local/lib/ruby/gems/2.6.0/bin/compass /usr/local/opt/ruby/bin/compass

8. Java 11

   brew install openjdk@11
   # Add to your PATH in terminal profile, i.e. ~/.bash_profile or ~/.zshrc
   export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

9. Elasticsearch 5.x

   # Download tar: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.0.tar.gz
   
   # Decompress
   tar -xvf ~/Downloads/elasticsearch-5.6.0.tar.gz -C /usr/local/opt/
   
   # Add to PATH in your terminal profile, i.e. ~/.bash_profile or ~/.zshrc
   export PATH="/usr/local/opt/elasticsearch-5.6.0/bin:$PATH"
   
   # Restart terminal and check versions
   elasticsearch -V

10. Brew dependencies

    brew install libmagic nginx graphviz redis

11. Chrome driver for Testing

    Chromedriver is needed in your PATH. If working in a python virtual environment, then the chromedriver can be added to your-venv-dir/bin directory.

    You also need to install Chrome (if not already installed). In addition, allow chromedriver (System Preferences->Security & Privacy) to run to run bdd tests

Application Installation

1. Create a virtual env in your work directory. Here we use python3 venv module. Use venv, like conda, if you please

   cd your-work-dir
   python3 -m venv snovault-venv
   source snovault-venv/bin/activate
   pip install -U pip==21.0.1

2. Clone the repo and install requirements

   cd snovault
   pip install -e '.[dev]'

   If psycopg2 fails to compile, you may need to set LDFLAGS to the output of pg_config --ldflags before pip installation.

   LDFLAGS=$(pg_config --ldflags) pip install -e '.[dev]'

   If you have errors at runtime that look like this:

   ImportError: dlopen(/Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so, 2): Symbol not found: _PQencryptPasswordConn
   Referenced from: /Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so
   Expected in: /usr/lib/libpq.5.dylib
   in /Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so

   you may need to add the brew-installed Postgres headers, usually -L/usr/local/opt/postgresql@11/lib, to the LDFLAGS in addition to the ones given by pg_config --ldflags.

3. Build Application

   # Make sure you are in the snovault-venv
   make clean && buildout

4. Run Application

   # Make sure you are in the snovault-venv
   dev-servers development.ini --app-name app --clear --init --load
   # In a separate terminal, make sure you are in the snovault-venv
   pserve development.ini

5. Browse to the interface at http://localhost:6543

6. Run Tests

   • no argument runs non bdd tests

   # Make sure you are in the snovault-venv
   ./circle-tests.sh bdd
   ./circle-tests.sh npm
   ./circle-tests.sh

Changes

Snovault 1.0.67 (released) encoded v129.2

1. SNO2-21-files-crash (#399)

Snovault 1.0.66 (released) encoded v129.0

1. ENCM-89-support-big-image-attachments (#390)

Snovault 1.0.65 (released) encoded v128.0

1. increase-field-limit (#388)

Snovault 1.0.64 (released) encoded v128.0

1. Pin splinter (#385)

Snovault 1.0.63 (released) encoded v128.0

1. ENCM-29-pin-antlr (#379)

Snovault 1.0.62 (released) encoded v124.0

1. SNO-217-register-search-config-pieces-from-item (#370)

Snovault 1.0.61 (released) encoded v124.0

1. SNO-214-upgrade-node-to-version-12-for-circleci-tests (#365)

2. SNO-210-read-columns-from-configs (#368)

3. SNO-216 Fix dev-servers 100% cpu usage when nothing to print. (#367)

4. SNO-212 Allow gzip attachments (#363)

5. SNO-213-circleci-node-install-fix (#364)

6. SNO-211-increase-batch-size-in-scroll-api (#360)

Snovault 1.0.60 (released) encoded v121.0

1. SNO-209-move-search-code-to-independent-package (#356)

2. SNO-206-use-redis-cache-for-rnget-view (#354)

3. SNO-207-facet-groups-support (#355)

4. SNO-205-allow-customization-of-default-sort (#353)

Snovault 1.0.59 (released) encoded v120.0

1. SNO-204-allow-facet-caching (#351)

Snovault 1.0.58 (released) encoded v118.0

1. SNO-170-avoid-rendering-from-database-3 (#346)

2. SNO-202-upgrade-python (#345)

Snovault 1.0.57 (released) encoded v116.0

1. revert SNO-202-python3.8.5-upgrade (#338)

2. SNO-201-add-search-config-registry (#336)

3. SNO-202-python3.8.5-upgrade (#335)

Snovault 1.0.56 (released) encoded v115.0

1. SNO-200-remove-nonsubstring-filter-from-mapping (#332)

2. SNO-199-allow-customization-of-search-fields (#333)

3. SNO-197-add-range-queries-and-min-max-aggs (#329)

4. SNO-198-allow-configuration-of-reserved-search-params (#330)

5. SNO-138-fix-pytest-warnings (#324)

6. SNO-182-readme-update (#323)

Snovault 1.0.55 (released) encoded v114.0

1. SNO-192-update-pip-install (#321)

Snovault 1.0.54 (released) encoded v113.0

1. SNO-27-pip-installable (#316)

2. SNO-191-pyramid-env-vars (#318)

3. SNO-117-search-as-you-type (#220)

Snovault 1.0.53 (released) encoded v112.0

1. SNO-158-create-minimally-indexed-deploy

2. SNO-190-pin-setuptools-in-circle-ci (#315)

Snovault 1.0.52 (released) encoded v111.0

1. SNO-185-Log-connection-error (#311)

2. SNO-183-allow-specification-of-cart-in-search (#310)

3. SNO-181-Update-install-readme (#309)

Snovault 1.0.51 (released) encoded v109.0

1. SNO-174-add-local-storage-redis (#299)

2. SNO-176-invalidate-object-when-unlinked (#303)

3. SNO-177-add-view-with-specified-fields (#300)

4. SNO-178-fix-indexer-test (#304)

Snovault 1.0.50 (released) encoded v108.0

1. SNO-168-stream-heterogeneous-uuids (#292)

2. SNO-169 Creating a command for admin user creation (#293)

Snovault 1.0.49 (released) encoded v106.0

1. SNO-166-update-query-string (#288)

Snovault 1.0.48 (released) encoded v105.0

1. SNO-163-install-graphviz (#285)

Snovault 1.0.47 (released) encoded v104.0

1. SNO-159-allow-remote-indexing (#282)

Snovault 1.0.46 (released) encoded v103.0 (unreleased)

1. SNO-160-use-boolean-values-in-facets (#280)

2. SNO-116 Use summary_matrix instead of summary (#278)

3. SNO-157-make-facets-collapsable (#275)

Snovault 1.0.45 (released) encoded v102.0

1. SNO-154-fix-magic-json-bug (#272)

2. SNO-156 Use pyramid_retry 2.1.1 (#273)

3. SNO-137-add-search-doc (#267)

4. SNO-152 fix circle-ci browser install (#269)

Snovault 1.0.44 (released) encoded v99.0

1. SNO-150 Remove duplicates and fix antlr version (#266)

2. SNO-149-allow-specification-of-audit-field (#265)

3. SNO-143-update-and-switch-readme

4. SNO-148-uuid-indexing-times-new-line-bit

5. SNO-144-redirect-downloads-through-nginx-or-apache (#192)

Snovault 1.0.43 (released) encoded v98.0

1. SNO-146-clean-up-threadmanager-on-pyramid-retry-exception (#261)

Snovault 1.0.42 (released) encoded v97.0

1. SNO-145-handle-missing-obj-on-request (#258)

Snovault 1.0.41 (released) encoded v97rc

1. SNO-142 Minor fixes for ubutnu 18 on v96x1rc1 (#252)

2. SNO-141-update-osx-catalina-readme (#252)

3. SNO-140 Allow shorting uuids with flag (#249)

4. SNO-139 Log detailed initial indexing times (#249)

5. SNO-135 Update installation for osx catalina (#249)

6. SNO-132 Update to py37 lrowe pr with fixes (#249)

Snovault 1.0.40 (released) encoded v95.0

1. SNO-129-add-fielded-generator-response (#244)

2. SNO-128-make-select-calculated-properties-view (#245)

3. SNO-133-setuptools-test-failure (#247)

Snovault 1.0.39 (released) encoded v94.0

1. SNO-126-allow-default-value-for-missing-aggregation-field (#242)

Snovault 1.0.38 (released) encoded v93.0

1. SNO-125-switch-batch-download-to-query-string (#238)

2. SNO-118-simple-query-string (#240)

3. SNO-123-limit-es-scan (#237)

4. SNO-121-fix-circleci-apt-install (#224)

Snovault 1.0.37 (released) encoded v92.0

1. SNO-119-escape-regex-slash (#221)

2. [HOTFIX] SNO-120-stream-large-json (#222)

3. SNO-115-delete-old-search-code (#219)

Snovault 1.0.36 (released) encoded v91.0

1. SNO-112-fix-npm-audit-issues (#216)

2. SNO-113-make-audit-view (#217)

Snovault 1.0.35 (UNRELEASED) encoded

1. SNO-95-refactor-search (#214)

Snovault 1.0.34 (released) encoded v88.0

1. SNO-108-improve-search-new (#212)

2. SNO-105-fix-batchupgrade-tests-sno103 (#211)

3. SNO-106-fix-spec-char-500-error (#209)

Snovault 1.0.33 (released) encoded v87.0

1. SNO-103 fix-batch-upgrade-error-logging (#206)

2. SNO-102 use-openjdk-java-for-circleci (#205)

Snovault 1.0.32 (released) encoded v87rc1

1. SNO-99 port travis tests to circleci (#202)

2. SNO-100 Parallelize batchupgrade script (#201)

3. SNO-92 Fix batch upgrade logging (#201)

4. SNO-101 Refactor batchupgrade (#201)

Snovault 1.0.31 (released) encoded v86.0

1. SNO-98 Update sauce connect to 4.5.3 (#199)

2. SNO-93 user-count-does-not-match (#191)

3. SNO-90 added-advancedQuery (#183)

Snovault 1.0.30 (released) encoded v85x2

1. SNO-96 fix indexing tests (#196)

Snovault 1.0.29 (released) encoded v85rc1

1. SNO-91 update-travis-java-ref (#188)

2. SNO-87 fixed-issue-while-profile-url-does-not-work (#179)

3. SNO-86 escape-search-text (#178)

4. SNO-88 fix-user-search-count (#180)

Snovault 1.0.28 (released) encoded v84rc1

1. SNO-89 Fix backoff error key in indexer (#181)

2. SNO-85 facet-reappearing (#175)

Snovault 1.0.27 (released) encoded v83.0

1. SNO-83 Rotate img attachment for EXIF (#174)

Snovault 1.0.26 (released) encoded v82.0

1. SNO-78 add long facet type (#168)

2. SNO-80 typeahead for facets (#168)

3. SNO-73 Add uuid queue module (#162)

Snovault 1.0.25 (released) encoded v81.0

1. SNO-77 Add review to submit_for validation (#166)

Snovault 1.0.24 (released) encoded v80.0

1. SNO-75-fix-linux-travis-option (#163)

Snovault 1.0.23 (released) encoded v79.0 also

1. SNO-72 Update simple for uuid module (#158)

Snovault 1.0.22 (released) encoded v79.0 also

1. SNO-68 Stop using _all for indexing (#155)

2. SNO-74 Remove npm shrinkwrap json (#157)

Snovault 1.0.21 (released) encoded v78.0 also

1. SNO-65 Refactor indexer uuids as server client (#151)

Snovault 1.0.20 (released) encoded v78.0

1. SNO-63 Update pip requests and remove wal-e reqs (#150)

2. SNO-66 Add new endpoint to map schema to schema titles (#152)

Snovault 1.0.19 (released) encoded v77.0

1. SNO-60-check-report-res-has-view (#147)

2. SNO-50 Initial shopping cart (#142)

3. SNO-59-fix-index-logger-name (#137)

4. SNO-53 Add index flags to indexers (#137)

Snovault 1.0.18 (released) encoded v76.0

1. SNO-49 Change audit inherit default (#132)

2. SNO-31 Refactor search related views (#141) (#143)

1.0.17 1. [HOTFIX] SNO-54-fix-schema-copy-line (#136)

1.0.16 1. SNO-52-alter-select-distinct-values (#131)

1.0.15 1. SNO-48-add-embed-cache-to-ini (#127)

1.0.14 1. SNO-45 Increase embed_capacity (#123)

1.0.13 1. SNO-46 Remove unused search type arg (#122) 2. SNO-43 Clean up snovault startup (#116)

1.0.12 1. SNO-42 Check call count explicitly (#118)

1.0.11 01. SNO-41-put-validator-accession (#112)

1.0.10 01. SNO-35 fix bug in get_rev_links(#111) 02. SNO-40 Upgrade blob storage to boto3 (#110)

1.0.9: 01. SNO-38 Return lists from get_related_uuids (#108, #105)

1.0.8: Released with 1.0.9 01. SNO-36-update-buildout (#104) 02. SNO-34-nginx-dev-proxy-headers (#103)

1.0.7: The only update was reverted. Empty Release.

1.0.6: 01. SNO-33 specify index for get_by_unique_key from collection (#94) 02. SNO-28 limit ES storage to indices created for snovault resources (#93)

1.0.5: 01. SNO-30 Split Indexer State from indexer file and update 02. SNO-10 Remove audit indexing via 2-pass 03. SNO-9 Add api end points to new indexer meta objects 04. SNO-25 Make uniqueItems to check the serialized values (#85) 05. SNO-26 Add schemas map view to profiles (#86) 06. SNO-29 Limit audits on form update (#87) 07. SNO-19 Update DOI preferred resolver url (#80)

1.0.4: -SNO-15 Add index to storage propertysheet -SNO-14 Update delete script

1.0.3: -SNO-8 Add JSONSchemas type to profiles page (#73)

1.0.2: * Issues discovered while release of ENCD v65 part 2

-SNO-12: Set max clause parameter in es indexer search #75 -SNO-11: Add timeout to ES indexer search query #74

1.0.1: * Issues discovered while release of ENCD v65

-SNO-6: Fix index query too many clauses failure -SNO-5: Update psycopg to match encoded version 2.7.3

1.0.0:

• 31 delete

• ES5 Fix: Update index settings shard number

0.33:

• ES5 Update: ENCD-2488 ES5 Update aka RM3910

• Fix travis build: Pre Install setuptools with pip for travis like ENCD-3722

0.32:

• Update to ENCD-3669 to not include notSubmittable reverse links in the edit view of an object.

0.31:

• ENCD-3684 Specify https index to fix buildout, update changelog.

• Use notSubmittable instead of calculatedProperty to indicate properties that may not be submitted. Related to ENCD-3669.

0.30:

• Document dependency on java 8.

• Disable 2nd indexing pass.

0.29:

• Fix recording indexing errors.

• Add some documentation about indexing.

0.28:

• Add support for adding and updating child objects specified as abstract types in the schema.

• Split indexing into 2 phases.

0.27:

• Move embed cache to connection and increase size.

• Fix reporting upgrade errors when error path includes an integer.

0.26:

• Indexer: Limit workers to 1 task and scale chunk size based on number of items being indexed.

0.25:

• Indexer: Limit workers to 4 tasks to avoid out-of-memory errors.

0.24:

• If the schema specifies an explicit mapping, use it when building the elasticsearch mapping. This provides an escape valve for edge cases (such as not indexing the layout structure of a page).

• upgrade to sauceconnect v4.4.4

• add port_range to wsgi_tests (mrmin)

0.23: - replace copy.deepcopy() for faster indexing

0.22: - New version of image magic, fix sauce labs

0.21: - (pypi errors, identical to 0.22)

0.20: - Patch mpindexer for better error messages