ws-availability 1.0.4

Add your description here

FDSNWS-Availability Deployment

Overview

• WFCatalog DB (gray) - database used by the WFCatalog collector and API.
• FDSNWS-Availability API (blue) - Flask-based FDSNWS-Availability implementation.
• FDSNWS-Availability Cache (green) - Redis-based cache to store restriction information.
• FDSNWS-Availability Cacher (orange) - Python-based container to harvest and store restriction information.
• FDSNWS-Availability Update (purple) - JS script to fill the availability materialized view using WFCatalog daily_streams and c_segments collections.

Following implementation requires MongoDB v4.2 or higher.

Deployment

1. Clone the [https://github.com/EIDA/ws-availability] repository and go to its root

2. Copy config.py.sample to config.py and adjust it as needed (please notice there are two sections - RUNMODE == "production" and RUNMODE == "test"; for Docker deployment use the production section):

   # WFCatalog MongoDB
   MONGODB_HOST = "localhost" #MongoDB host
   MONGODB_PORT = 27017 #MongoDB port
   MONGODB_USR = "" #MongoDB user
   MONGODB_PWD = "" #MongoDB password
   MONGODB_NAME = "wfrepo" #MongoDB database name
   FDSNWS_STATION_URL = "https://orfeus-eu.org/fdsnws/station/1/query" #FDSNWS-Station endpoint to harvest restriction information from
   CACHE_HOST = "localhost" #Cache host
   CACHE_PORT = 6379 #Cache port
   CACHE_INVENTORY_KEY = "inventory" #Cache key for restriction information
   CACHE_INVENTORY_PERIOD = 0 #Cache invalidation period for `inventory` key; 0 = never invalidate
   CACHE_RESP_PERIOD = 1200 #Cache invalidation period for API response
   

3. Build the containers:

   docker-compose -p 'fdsnws-availability' up -d --no-deps --build
   

   When the Docker stack is deployed, you will see 3 containers running:

   $ docker ps
   CONTAINER ID   IMAGE                         COMMAND                  CREATED         STATUS        PORTS                      NAMES
   4e3dace01fb0   fdsnws-availability_api       "/bin/bash -c 'gunic…"   10 seconds ago  Up 5 seconds  0.0.0.0:9001->9001/tcp     fdsnws-availability-api
   3c91e0d1c5e6   fdsnws-availability_cacher    "/bin/bash -c 'pytho…"   10 seconds ago  Up 5 seconds  0.0.0.0:11211->11211/tcp   fdsnws-availability-cacher
   d983e64d64a8   redis:7.0-alpine              "docker-entrypoint.s…"   10 seconds ago  Up 5 seconds  0.0.0.0:6379->6379/tcp     fdsnws-availability-cache
   

   You can follow the fdsnws-availability-cacher container to see the status of restriction information harvesting:

   $ docker logs --follow fdsnws-availability-cacher
   [2023-01-11 09:47:38 +0000] [0] [INFO] Getting inventory from FDSNWS-Station...
   [2023-01-11 09:47:39 +0000] [0] [INFO] Harvesting 33 from https://orfeus-eu.org/fdsnws/station/1/query?level=network: 2M,3T,6A...
   #...
   [2023-02-15 08:31:56 +0000] [0] [INFO] Completed caching inventory from FDSNWS-Station
   

   Once fdsnws-availability-cacher is completed, it will go down. Harvested information is stored in the Redis DB served by fdsnws-availability-cache container. To rebuild the cache, simply restart the container using:

   docker start fdsnws-availability-cacher
   

   To automate cache rebuilding process, add following line to cron:

   # Rebuild FDSNWS-Availability restriction information cache daily at 3:00 AM
   0 3 * * * docker restart fdsnws-availability-cacher
   

   It will harvest and overwrite the restricted information stored in Redis instance.

4. Materialized view

   1. Initial build

      When the stack is initially deployed, the materialized view is not yet in place. To build it, issue the following command:

      # Script started on 2023-02-24
      $ mongosh -u USER -p PASSWORD --authenticationDatabase wfrepo --eval "daysBack=365" views/main.js
      Processing WFCatalog entries using networks: '^.*$', stations: '^.*$', start: '2022-03-24', end: '2023-03-24' completed!
      

      It will go throught the documents in daily_streams and c_segments from last year, extract availability information and store it in the availability materialized view.

   2. Daily appension

      To automate availability information appension, add following line to cron:

      0 6 * * * cd ~/ws-availability/views && mongosh -u USERNAME -p PASSWORD --authenticationDatabase wfrepo main.js > /dev/null 2>&1
      

      It will go throught the documents in daily_streams and c_segments from last day, extract availability information and append it to the availability materialized view. If additional parameters are not provided, script processes data from last day:

      # Script started on 2023-02-24
      $ mongosh -u USERNAME -p PASSWORD --authenticationDatabase wfrepo main.js
      Processing WFCatalog entries using networks: '^.*$', stations: '^.*$', start: '2023-03-23', end: '2023-03-24' completed!
      

   3. Back-processing

      Processing can be also executed on a predefined subset of data using networks, stations, start and end parameters.

      # Last week
      $ mongosh -u USERNAME -p PASSWORD --authenticationDatabase wfrepo --eval "daysBack=7;" main.js
      Processing WFCatalog entries using networks: '^.*$', stations: '^.*$', start: '2023-03-17', end: '2023-03-24' completed!
      
      # January 2023
      $ mongosh -u USERNAME -p PASSWORD --authenticationDatabase wfrepo --eval "start='2023-01-01'; end='2023-01-31'" main.js
      Processing WFCatalog entries using networks: '^.*$', stations: '^.*$', start: '2023-01-01', end: '2023-01-31' completed!
      
      # NL.HGN data between December 2022 and January 2023
      $ mongosh -u USERNAME -p PASSWORD --authenticationDatabase wfrepo --eval "networks='NL'; stations='HGN'; start='2022-12-01'; end='2023-01-31'" main.js
      Processing WFCatalog entries using networks: '^NL$', stations: '^HGN$', start: '2022-12-01', end: '2023-01-31' completed!
      
      # You can also use regular expressiosn for `networks` and `stations` params
      # Please refer to [docs](https://www.mongodb.com/docs/manual/reference/operator/query/regex/) for details
      
      # Stations from NL network matching `G*4` template with timespan from 2023-03-01 till 2023-03-02
      $ mongosh -u USERNAME -p PASSWORD --authenticationDatabase wfrepo --eval "networks='NL'; stations='G.*4'; start='2023-03-01'; end='2023-03-02'" main.js
      Processing WFCatalog entries using networks: '^NL$', stations: '^G.*4$', start: '2023-03-01', end: '2023-03-02' completed!
      
      # Stations from `NL` or `NA` networks with station codes `HGN` or `SABA` and timespan from 2023-03-01 till 2023-03-02
      $ mongosh -u USERNAME -p PASSWORD --authenticationDatabase wfrepo --eval "networks='NL|NA'; stations='HGN|SABA'; start='2023-03-01'; end='2023-03-02'" main.js
      Processing WFCatalog entries using networks: '^NL|NA$', stations: '^HGN|SABA$', start: '2023-03-01', end: '2023-03-02' completed!
      
      # All stations from networks `NL` and `NA` with timespan from 2023-03-01 till 2023-03-02
      $ mongosh -u jarek -p password123 --authenticationDatabase wfrepo --eval "networks='NL|NA'; start='2023-03-01'; end='2023-03-02'" main.js
      Processing WFCatalog entries using networks: '^NL|NA$', stations: '^.*$', start: '2023-03-01', end: '2023-03-02' completed!
      

   4. Indexes

      It is highly suggested to create at least following index in the availability materialized view. First, login to your MongoDB instance using mongosh and then execute following commands:

      use wfrepo;
      db.availability.createIndex({ net: 1, sta: 1, loc: 1, cha: 1, ts: 1, te: 1 })
      

5. Validation

   Now it is time to check if everything is running (remember to change the net query parameter). API is exposed by default on port 9001, let's try to get the landing page:

   $ curl "127.0.0.1:9001"
   <!DOCTYPE html>
   <html lang="en">
     <head>
       <meta charset="utf-8" />
       <meta name="author" content="gempa GmbH" />
       <title>FDSNWS-Availability</title>
     </head>
     <body>
       <h1>FDSNWS Availability Web Service</h1>
       <p>
         The availability web service returns detailed time span information about
         available time series data. Please refer to
         <a href="http://www.fdsn.org/webservices"
           >http://www.fdsn.org/webservice</a
         >
         for a complete service description.
       </p>
   
       <h2>Available URLs</h2>
       <ul>
         <li><a href="query">query</a></li>
         <li><a href="extent">extent</a></li>
         <li><a href="version">version</a></li>
         <li><a href="application.wadl">application.wadl</a></li>
       </ul>
     </body>
   

   Get request to the /extent method:

   $ curl "127.0.0.1:9001/extent?net=NA&start=2023-02-01"
   #Network Station Location Channel Quality SampleRate Earliest                    Latest                      Updated              TimeSpans Restriction
   NA       SABA             BHE     D       40.0       2023-02-01T00:00:00.000000Z 2023-02-14T00:00:00.000000Z 2023-02-14T07:41:14Z 1         OPEN       
   NA       SABA             BHN     D       40.0       2023-02-01T00:00:00.000000Z 2023-02-14T00:00:00.000000Z 2023-02-14T07:42:07Z 1         OPEN       
   NA       SABA             BHZ     D       40.0       2023-02-01T00:00:00.000000Z 2023-02-14T00:00:00.000000Z 2023-02-14T07:41:41Z 1         OPEN       
   # ...
   

   Get request to the /query method:

   $ curl "127.0.0.1:9001/query?net=NA&start=2023-02-01"
   #Network Station Location Channel Quality SampleRate Earliest                    Latest                     
   NA       SABA             BHE     D       40.0       2023-02-01T00:00:00.000000Z 2023-02-02T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-02T00:00:00.000000Z 2023-02-03T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-03T00:00:00.000000Z 2023-02-04T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-04T00:00:00.000000Z 2023-02-05T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-05T00:00:00.000000Z 2023-02-06T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-06T00:00:00.000000Z 2023-02-07T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-07T00:00:00.000000Z 2023-02-08T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-08T00:00:00.000000Z 2023-02-09T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-09T00:00:00.000000Z 2023-02-10T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-10T00:00:00.000000Z 2023-02-11T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-11T00:00:00.000000Z 2023-02-12T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-12T00:00:00.000000Z 2023-02-13T00:00:00.000000Z
   NA       SABA             BHE     D       40.0       2023-02-13T00:00:00.000000Z 2023-02-14T00:00:00.000000Z
   

6. Reverse proxy example config

   An example of Apache reverse proxy config:

   # FDSNWS-Availability (Docker)
   <Location /fdsnws/availability/1>
     # in order to omit CORS error
     Header add Access-Control-Allow-Origin "*"
   </Location>
   ProxyPass /fdsnws/availability/1 <HOST>:9001 timeout=600
   ProxyPassReverse /fdsnws/availability/1 <HOST>:9001 timeout=600
   

Performance Tuning

Gunicorn Workers Configuration

The number of Gunicorn workers directly affects how many concurrent requests your service can handle. The default configuration uses 1 worker for maximum stability on resource-constrained servers.

Current Configuration (docker-compose.yml)

command: gunicorn --bind 0.0.0.0:9001 --workers 1 start:app

Adjusting Worker Count

For servers with limited resources or thread creation issues:

# Minimum configuration (most stable)
command: gunicorn --bind 0.0.0.0:9001 --workers 1 --timeout 600 start:app

For servers with moderate resources:

# 2-3 workers (recommended for most deployments)
command: gunicorn --bind 0.0.0.0:9001 --workers 2 --timeout 600 start:app

For high-performance servers:

# Formula: (2 × CPU cores) + 1
# Example for 4-core server: --workers 9
command: gunicorn --bind 0.0.0.0:9001 --workers 4 --timeout 600 start:app

Important Notes

1. Each worker is a separate process with its own memory footprint

2. More workers ≠ always better - too many workers can exhaust system resources

3. Monitor for errors after increasing workers:

   docker logs -f fdsnws-availability-api
   # Watch for "pthread_create failed" or similar errors
   

4. Resource usage check:

   docker stats fdsnws-availability-api
   # If CPU < 80% and memory available, you can add more workers
   

MongoDB Connection Pool

The MongoDB connection pool is configured in apps/wfcatalog_client.py:

maxPoolSize=1  # Connections per worker

How It Works

• Each Gunicorn worker has its own MongoDB client
• Total connections = workers × maxPoolSize
• Example: 2 workers × 1 pool = 2 total MongoDB connections

When to Adjust

Keep maxPoolSize=1 if:

• ✅ Using sync workers (default Gunicorn configuration)
• ✅ Each worker handles one request at a time
• ✅ Server has resource constraints

Increase maxPoolSize only if:

• Using async workers (gevent/eventlet)
• Using threading within workers
• MongoDB is a bottleneck (check with profiling)

Example Configurations

Workers	maxPoolSize	Total Connections	Use Case
1	1	1	Minimal (default)
2	1	2	Recommended
4	1	4	High performance
2	5	10	Async workers

Thread Limiting (Important!)

The configuration includes thread limits to prevent pthread_create failed errors on restricted servers:

environment:
  OPENBLAS_NUM_THREADS: 1
  MKL_NUM_THREADS: 1
  NUMEXPR_NUM_THREADS: 1
  OMP_NUM_THREADS: 1

Do not remove these unless you're certain your server can handle multiple threads per process. These prevent NumPy/ObsPy from spawning excessive threads.

Troubleshooting

Problem: Service crashes with "pthread_create failed"

• Solution: Reduce workers to 1, keep thread limits in place

Problem: Slow response times under load

• Solution: Increase workers (if resources allow), monitor with docker stats

Problem: High memory usage

• Solution: Reduce workers, check for memory leaks with profiling

Problem: MongoDB connection errors

• Solution: Check total connections (workers × maxPoolSize) against MongoDB limits

Performance Monitoring

See tests/performance/ for profiling and benchmarking tools:

# Quick performance test
bash tests/performance/quick_test.sh

# Detailed profiling
python tests/performance/profiler.py

# Load testing
locust -f tests/performance/locustfile.py --host=http://localhost:9001

For more details, see Performance Analysis Plan.

Running in development environment

1. Go to the root directory.

2. Copy config.py.sample to config.py and adjust it as needed.

3. Create the virtual environment:

   python3 -m venv env
   

4. Activate the virtual environment:

   source env/bin/activate
   

5. Install the dependencies:

   pip install -r requirements.txt
   

6. Create Redis instance (mandatory for WFCatalog-based deployment):

   docker run -p 6379:6379 --name cache -d redis:7.0-alpine redis-server --save 20 1 --loglevel warning
   

7. Build the cache:

   python3 cache.py
   

8. Now you can either:

   1. Run it:

      RUNMODE=test FLASK_APP=start.py flask run
      
      # Or with gunicorn:
      RUNMODE=test gunicorn --workers 2 --timeout 60 --bind 0.0.0.0:9001 start:app
      

   2. Debug it in VS Code (F5) after selecting "Launch (Flask)" config.

RUNMODE builtin values

• production
• test

Tests

Tests can be executed from the respository root using following command:

PYTHONPATH=./apps/ python3 -m unittest discover tests/

Ideas for improvements

1. Move restriction information from Redis cache directly to the db.availability materialized view. This would imply modifying the views/main.js script with code harvesting this information directly from the FDSNWS-Station instance.
2. Modify underlying RESIF code from logic based on list of arrays to list of objects/dicts which is native MongoDB response to prevent the object/dict to array casting.

References

This repository has been forked from gitlab.com/resif/ws-availability, special thanks to our colleagues at RESIF for sharing their implementation of the FDSNWS-Availability web service. 💐