django-seeding 1.3.6

Simple Django Package that helps developer to seed data from files and codes into the database automatically

Django Seeding

Table of Contents

• Introduction
• Installation
• Simple Example
• Full Usage Documentation
  • Seeders List
  • Attributes List
  • Run Methods
• Built-in RESTfull APIs for Management
• Full Examples
  • CSVFileModelSeeder (Recommended)
  • JSONFileModelSeeder (Recommended)
  • JSONFileChildSeeder (Recommended)
  • CSVFileSerializerSeeder
  • JSONFileSerializerSeeder
  • EmptySeeder (Recommended)
  • ModelSeeder (Recommended)
  • SerializerSeeder
  • Seeder
• Contributing
• Contact
• License

Introduction

This package helps developers to fill the database with real data instead of filling it manually.

Data can be presented as CSV File , JSON File or in-code.

Dependency-Injection also available to inject your logic by specifying a serializer_class or writing your custom seed method.

Installation

Installing using pip:

pip install django-seeding:

add 'django_seeding' to your INSTALLED_APPS setting:

INSTALLED_APPS = [
    ...
    'django_seeding',
]

Simple Example

Let's take a look at a quick example of using CSVFileModelSeeder seeder from django-seeding to build a simple seeder to insert data in the database.

django_seeding_example/models.py:

from django.db import models


class M1(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()

django_seeding_example/seeders.py:

from django_seeding import seeders
from django_seeding.seeder_registry import SeederRegistry 
from django_seeding_example.models import M1

@SeederRegistry.register
class M1Seeder(seeders.CSVFileModelSeeder):
    model = M1
    csv_file_path = 'django_seeding_example/seeders_data/M1Seeder.csv'

django_seeding_example/seeders_data/M1Seeder.csv:

title,description
t1,d1
t2,d2

Now you just need to run this command:

python manage.py seed

Full Usage Documentation:

Now lets go deeper into the different Seeders types with its details:

Seeders List:

• CSVFileModelSeeder (Recommended)
• JSONFileModelSeeder (Recommended)
• JSONFileChildSeeder (Recommended)
• CSVFileSerializerSeeder
• JSONFileSerializerSeeder
• EmptySeeder (Recommended)
• ModelSeeder (Recommended)
• SerializerSeeder
• Seeder

Attributes List

In general there is a way to know how to deal with these seeders easily:

Model..Seeder needs model class-attribute

Serializer..Seeder needs serializer_class class-attribute

CSVFile..Seeder needs csv_file_path class-attribute

JSONFile..Seeder needs json_file_path class-attribute

All seeders can takes these optional class-attributes:

• id: str (So Recommended)

  This is what will be stored in the AppliedSeeder table to check if a seeder is already applied or not

  It is recommended to set it as the seeder name

  So, set it and don't change it because when the value is changed it will be considerd as a new seeder and it will be applied again even that the old seeder with the old name is applied

  default value: str(type(seeder))

• priority: int|float

  Seeders will be sorted depending on this attribute (lower-first)

  default value: float('inf')

• just_debug: bool

  This attribute specify if the seeder will be applied when the server is in the production-mode or not depending in the DEBUG variable in settings file

  DEBUG=False & just_debug=True -> don't apply

  DEBUG=False & just_debug=False -> apply

  DEBUG=True & just_debug=False -> apply

  DEBUG=True & just_debug=True -> apply

  default value: False

Notice:

• @SeederRegistry.register is the decorator that register the seeder, so, if this decorator is not applied then the seeder will not be applied

• Model seeders use bulk_create method, so, they are faster than Serializer seeders

• Child seeders use bulk_create method with caching fetch for related objects, so, they are faster than Serializer seeders

• CSV file reader is using pandas for a better performance and less bugs

• Using Model seeders means the field names must match the fields you have defined in your model

• Using Serializer seeders means the field names must match the fields you have defined in your serializer

• you can define get_ class-methods instead of class-attributes as below:

  • get_model
  • get_serializer_class
  • get_csv_file_path
  • get_json_file_path
  • get_id
  • get_priority
  • get_just_debug

Run methods:

• To seed with a manual command (Recommended):

python manage.py seed

• To seed with runserver manually just add "--seed" in runserver command:

python manage.py runserver --seed

• To seed on runserver automatically just set in your project settings:

SEEDING_ON_RUNSERVER = True

Automatically Seeding:

• If you set SEEDING_ON_RUNSERVER=True in your settings file You can stop seeding in a runserver by using --dont-seed argument

python manage.py runserver --dont-seed

Managing Debug Mode with the seed Command:

By default, the seed command will use the DEBUG setting from your Django project's settings.py. However, you can override this by explicitly passing the --debug option when running the command.

Force DEBUG to True:

python manage.py seed --debug=True

Force DEBUG to False:

python manage.py seed --debug=False

If no value is specified for --debug, the command will fall back to the project's current DEBUG setting.

Built-in RESTfull APIs for Management

To manage seeders from a dashboard, we provide APIs that offer full control over seeder operations.

Usage:

Route django-seeding views in our seeder app urls.py:

from rest_framework import routers
from django_seeding.apis.views import AppliedSeederViewSet, RegisteredSeederViewSet


router = routers.DefaultRouter()

router.register('registered-seeders', RegisteredSeederViewSet, 'registered-seeders')
router.register('applied-seeders', AppliedSeederViewSet, 'applied-seeders')

urlpatterns = router.urls 

Route our seeder app in the project urls.py:

from django.urls import include
...
urlpatterns = [
    ...
    path('seeder/', include('django_seeding_example.urls')),
    ...
]
...

Customization:

You can customize the permissions or any other logic by inheriting the provided viewsets and overriding the necessary methods to meet your specific project requirements.

Postman Collection for API Testing:

To make it easier for you to test the APIs, we have provided a Postman collection containing all the API endpoints. You can download the collection and import it into Postman for quick and easy testing.

Steps to Use the Postman Collection:

• Download the Postman collection from this link.
• Open Postman, click on "Import", and select the .json file you just downloaded.
• You will now have all the API endpoints pre-configured and ready for testing in Postman.

Registered Seeder Endpoint:

In this package, seeder classes must be registered to be applied. This endpoint allows you to:

• List all registered seeders: This includes both applied and non-applied seeders.

  GET /registered-seeders/

• Apply specific seeders: You can apply all seeders or select specific ones.

  POST /registered-seeders/seed-all/

  we can specify the debug mode for the seeding and the needed ids to be seeded in the body like this (both of them are optional - can be null):

  {
      "debug": true,
      "ids": [
          "id1",
          "id2"
      ]
  }
  

Applied Seeders Endpoint:

Applied seeders are tracked in a dedicated model to prevent them from being applied multiple times. The Applied Seeder endpoints provide full management over the applied seeder records, allowing you to:

• List all applied seeders: View all seeders that have been applied.

  Get /applied-seeders/

• Retrieve a specific applied seeder: View details of a particular seeder.

  Get /applied-seeders/{id}/

• Mark a seeder as applied: Create a record in the applied seeder model to indicate it has been applied.

  POST /applied-seeders/

  Request body:

  {
      "id": "id"
  }
  

• Update a seeder record: Useful for changing a seeder's ID.

  PUT /applied-seeders/{id}/

  Request body:

  {
      "id": "new_id"
  }
  

• Delete a specific applied seeder: Removes a record to mark the seeder as not applied.

  DELETE /applied-seeders/{id}/

• Delete all applied seeders: This allows all seeders to be re-applied.

  DELETE /applied-seeders/delete-all/

Applied Seeder Table Schema:

• id: int (Primary Key)

• created_at: datetime

• updated_at: datetime

Full Seeders Examples:

Here we will go deeper in the seeders classes and its details

CSVFileModelSeeder (Recommended):

Fast bulk_create seeder

notice that the titles in the csv-file have to match the field names in the model

models.py

class M1(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()

seeders.py

@SeederRegistry.register
class M1Seeder(seeders.CSVFileModelSeeder):
    id = 'M1Seeder'
    priority = 1
    model = M1
    csv_file_path = 'django_seeding_example/seeders_data/M1Seeder.csv'

seeders_data/M1Seeder.csv

title,description
t1,d1
t2,d2

JSONFileModelSeeder (Recommended):

Fast bulk_create seeder

notice that the keys in the json-file must match the field names in the model

models.py

class M2(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()

seeders.py

@SeederRegistry.register
class M2Seeder(seeders.JSONFileModelSeeder):
    id = 'M2Seeder'
    priority = 2
    model = M2
    json_file_path = 'django_seeding_example/seeders_data/M2Seeder.json'

seeders_data/M2Seeder.json

[
    {
        "title": "json t1",
        "description": "json d1"
    },
    {
        "title": "json t2",
        "description": "json d2"
    }
]

JSONFileChildSeeder (Recommended):

Blinky-fast bulk-create seeder implemented with caching strategy.

This seeder was conceived to seed child models, i.e. models that at least one field is a foreign key (models.ForeignKey), but can be used instead of JSONFileModelSeeder for general models as well.

Notice that the keys in the json-file must match the field names in the model and also the structure. Parent models are represented as inner dicts.

models.py

class Father(models.Model):
    name = models.TextField()

class Son(models.Model):
    name = models.TextField()
    father = models.ForeignKey(Father, on_delete=models.CASCADE)

seeders.py

@SeederRegistry.register
class SonSeeder(seeders.JSONFileChildSeeder):
    id = 'SonSeeder'
    model = Son
    priority = 10
    json_file_path = 'django_seeding_example/seeders_data/SonSeeder.json'

seeders_data/SonSeeder.json

[
    {
        "name": "json son 1",
        "father": { "name": "json father 1" }
    },
    {
        "name": "json son 2",
        "father": { "name": "json father 2" }
    }
]

Notice that child priority must be greater than parent priority in order to the parent model be seeded before. Not seeding parent before will raise errors! Each field that is a FK must be a dictionary with field names same as its related model.

This seeder class can handle pretty complex relations between models. Let's expand the family (pun intended):

models.py

class Mother(models.Model):
    name = models.TextField()

class Daughter(models.Model):
    name = models.TextField()
    father = models.ForeignKey(Father, on_delete=models.CASCADE)
    mother = models.ForeignKey(Mother, on_delete=models.CASCADE)

    class Meta:
        constraints = [
            UniqueConstraint (
                fields=['name', 'father', 'mother'],
                name='unique_parentage'
            )]

class Grandson(models.Model):
    name = models.TextField()
    parentage = models.ForeignKey(Daughter, on_delete=models.CASCADE)

seeders.py

@SeederRegistry.register
class DaughterSeeder(seeders.JSONFileChildSeeder):
    id = 'DaughterSeeder'
    priority = 10
    model = Daughter
    json_file_path = 'django_seeding_example/seeders_data/DaughterSeeder.json'


@SeederRegistry.register
class GrandsonSeeder(seeders.JSONFileChildSeeder):
    id = 'GrandsonSeeder'
    model = Grandson
    json_file_path = 'django_seeding_example/seeders_data/GrandsonSeeder.json'

seeders_data/DaughterSeeder.json

[
    {
        "name": "json daughter 1",
        "father": { "name": "json father 1" },
        "mother": { "name": "json mother 1" }
    },
    {
        "name": "json daughter 2",
        "father": { "name": "json father 2" },
        "mother": { "name": "json mother 2" }
    }
]

seeders_data/GrandsonSeeder.json

[
    {
        "name": "json grandson 1",
        "parentage": {
            "name": "json daughter 1",
            "father": { "name": "json father 1" },
            "mother": { "name": "json mother 1" }
        }
    },
    {
        "name": "json grandson 2",
        "parentage": {
            "name": "json daughter 2",
            "father": { "name": "json father 2" },
            "mother": { "name": "json mother 2" }
        }
    }
]

CSVFileSerializerSeeder:

Slow one-by-one seeder

notice that the titles in the csv-file have to match the field names in the serializer

This seeder is used to inject a serializer to implement custom create logic

models.py

class M3(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()

serializers.py

class M3Serializer(serializers.ModelSerializer):
    class Meta:
        model = M3
        fields = ['title', 'description']

    def create(self, validated_data):
        validated_data['title'] = '__' + validated_data['title'] + '__'
        validated_data['description'] = '__' + validated_data['description'] + '__'
        return super().create(validated_data)

seeders.py

@SeederRegistry.register
class M3Seeder(seeders.CSVFileSerializerSeeder):
    id = 'M3Seeder'
    priority = 3
    serializer_class = M3Serializer
    csv_file_path = 'django_seeding_example/seeders_data/M3Seeder.csv'

seeders_data/M3Seeder.csv

title,description
t1,d1
t2,d2

JSONFileSerializerSeeder:

Slow one-by-one seeder

notice that the keys in the json-file have to match the field names in the serializer

This seeder is used to inject a serializer to implement custom create logic

models.py

class M4(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()

serializers.py

class M4Serializer(serializers.ModelSerializer):
    class Meta:
        model = M4
        fields = ['title', 'description']

    def create(self, validated_data):
        validated_data['title'] = '__' + validated_data['title'] + '__'
        validated_data['description'] = '__' + validated_data['description'] + '__'
        return super().create(validated_data)

seeders.py

@SeederRegistry.register
class M4Seeder(seeders.JSONFileSerializerSeeder):
    id = 'M4Seeder'
    priority = 4
    serializer_class = M4Serializer
    json_file_path = 'django_seeding_example/seeders_data/M4Seeder.json'

seeders_data/M4Seeder.json

[
    {
        "title": "json t1",
        "description": "json d1"
    },
    {
        "title": "json t2",
        "description": "json d2"
    }
]

EmptySeeder (Recommended):

Fast bulk_create seeder

models.py

class M5(models.Model):
    title = models.CharField(max_length=100, null=True)
    description = models.TextField(null=True)

seeders.py

@SeederRegistry.register
class M5Seeder(seeders.EmptySeeder):
    id = 'M5Seeder'
    priority = 5
    model = M5
    records_count = 2

ModelSeeder (Recommended):

Fast bulk_create seeder

notice that the keys in the data class-attribute have to match the field names in the model

models.py

class M6(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()

seeders.py

@SeederRegistry.register
class M6Seeder(seeders.ModelSeeder):
    id = 'M6Seeder'
    priority = 6
    model = M6
    data = [
        {
            "title": "in-code t1",
            "description": "in-code d1"
        },
        {
            "title": "in-code t2",
            "description": "in-code d2"
        },
    ]

SerializerSeeder:

Slow one-by-one seeder

notice that the keys in the data class-attribute have to match the field names in the serializer

This seeder is used to inject a serializer to implement custom create logic

models.py

class M7(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()

serializer.py

class M7Serializer(serializers.ModelSerializer):
    class Meta:
        model = M7
        fields = ['title', 'description']

    def create(self, validated_data):
        validated_data['title'] = '__' + validated_data['title'] + '__'
        validated_data['description'] = '__' + validated_data['description'] + '__'
        return super().create(validated_data)

seeders.py

@SeederRegistry.register
class M7Seeder(seeders.SerializerSeeder):
    id = 'M7Seeder'
    priority = 7
    serializer_class = M7Serializer
    data = [
        {
            "title": "in-code t1",
            "description": "in-code d1"
        },
        {
            "title": "in-code t2",
            "description": "in-code d2"
        },
    ]

Seeder:

Here you can write your logic as you want in the seed method

models.py

class Post(models.Model):
    content = models.TextField()


class Comment(models.Model):
    post = models.ForeignKey(Post, on_delete=models.CASCADE)
    content = models.TextField()

seeders.py

@SeederRegistry.register
class CustomSeeder(seeders.Seeder):
    id = 'CustomSeeder'
    priority = 8
    
    def seed(self):
        post1 = Post.objects.create(content='post1')
        post2 = Post.objects.create(content='post1')

        comment1 = Comment.objects.create(post=post1, content='comment1')
        comment2 = Comment.objects.create(post=post1, content='comment2')
        comment3 = Comment.objects.create(post=post2, content='comment3')
        comment4 = Comment.objects.create(post=post2, content='comment4')

Contributing

If you have suggestions for how Django Seeding could be improved, or want to report a bug, open an issue! We'd love all and any contributions.

For more, check out the Contributing Guide.

Contact

Suliman Awad - [e-mail] - Linkedin

Project Link: https://github.com/suliman-99/django-seeding

License

MIT License

Copyright (c) 2023 Suliman Awad

For more, check out the License File.