SEEDS_MONGODB_USERNAME

A string of the username of the database.

SEEDS_MONGODB_PASSWORD

A string of the password of the database.

SEEDS_MONGODB_HOST

A string of the ip address or the domain of the database.

SEEDS_MONGODB_PORT

A int of the port of the database.

SEEDS_MONGODB_DATABASE

A string of the name of the database.

SEEDS_MONGODB_COLLECTION

A list of the indexes to create on the collection.

SEEDS_MONGODB_OPTIONS_*

Options can be attached when the seed loader start to connect to MongoBD.

If any options are needed, the option can be set with the prefix SEEDS_MONGODB_OPTIONS_, the pipeline will parse it.

For example:

For more options, please refer to the page:

• Connection String URI Format — MongoDB Manual 3.4

SEEDS_MONGODB_SEEDS_QUERY

A dictionary as the keyword arguments to query. The default value is:

SEEDS_MONGODB_SEEDS_QUERY = {
    'filter': None,
    'projection': None,
    'skip': 0,
    'limit': 0,
    'no_cursor_timeout': False,
    'cursor_type': CursorType.NON_TAILABLE,
    'sort': None,
    'allow_partial_results': False,
    'oplog_replay': False,
    'modifiers': None,
    'manipulate': True
}

The keys and values in this setting is followed the keyword arguments of the method find of collection in pymongo. Please refer to the following documents for more details.

• collection – Collection level operations — PyMongo 3.5.1 documentation

SEEDS_MONGODB_SEEDS_BATCH_SIZE

A int of The batch size that each query will return, the default value is 100.

SEEDS_MONGODB_SEEDS_PREPARE

A string of the module path to the function to process the task (seed) from MongoDB.

The input will be the document queried from the collection set in SEEDS_MONGODB_COLLECTION, and the output should be a iterator which will return tuples with two elements: (url, doc). The url will be the argument url of Request, and the doc will be given to request.meta. As an example, the default function in this middleware:

class MongoDBSeedLoader(SeedLoader):

    ...

    def open_spider(self, spider: Spider):
        try:
            if self.settings.get(SEEDS_MONGODB_SEEDS_PREPARE):
                self.prepare = load_object(
                    self.settings.get(SEEDS_MONGODB_SEEDS_PREPARE))
            else:
                self.prepare = lambda x: map(
                    lambda y: (y, {'seed': x}),
                    x['websites'])
        except:
            raise NotConfigured

        ...

    ...