Visitar URL original
Add initial set of tests for composite primary and foreign keys by tillspeicher · Pull Request #138 · ormar-orm/ormar · GitHub
Skip to content

Add initial set of tests for composite primary and foreign keys - #138

Closed
tillspeicher wants to merge 1 commit into
ormar-orm:masterfrom
tillspeicher:composite_pk_fk
Closed

tillspeicher wants to merge 1 commit into
ormar-orm:masterfrom
tillspeicher:composite_pk_fk

Conversation

@tillspeicher

Copy link
Copy Markdown

I started this PR for the composite PK and FK implementation, please don't merge it yet.
So far, I only added a set of tests that covers the interfaces and behavior that we discussed in #137 . The tests likely contain errors, because there is no implementation of the actual functionality yet, so I couldn't run the code.
There are probably also some missing corner cases that should be tested as well.

For now, this is meant primarily as a specification for the composite PK and FK feature and to document the requirements.
In particular, the tests demonstrate and check composite PK- and FK-based CRUD operations, preventing direct access to FK columns, checking that reverse and many-to-many relationships are created correctly, that composite FKs are compatible with pydantic and that the pk property works as discussed.
In addition to what we discussed before, I added a test to ensure that composite PKs cannot use FK columns directly, as that would expose them via the pk property. So they need to use the names of the FK relations instead.

Please let me know if you would like anything to be changed or added @collerek . Otherwise I'll go ahead with the implementation.

database = database

id: int = ormar.Integer(primary_key=True)
assigned_by: Optional[User] = ormar.ForeignKey(User)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now you cannot declare relationship in Through model -> that's one of the things I wanted to add gradually, but for now ormar creates the fields during expanding_reverse_relations, relations are not copied in inheritance, and explicitly stating any relation on through Model will raise exception.

The key is that you can have multiple relations between two models, and ormar wouldn't know if the relation you created is the one that should be used for m2m relation or it should create other field for this relation (relation field name would have to be passed to ManyToMany probably).

@collerek

Copy link
Copy Markdown
Collaborator

Ok, in general the tests look good, as you said those test are responsible for checks regarding Model declaration, especially getting/ setting pks.

Since pk is such a crucial thing that is used almost everywhere (among others: order by, joins, filters, relations etc.), tests for this have to pretty much cover everything. Some of the following tests have common parts, but for clarity I listed what should be tested.

That includes, but is not limited to:

  • All Model methods (save/update/upsert/load etc.)
  • All QuerySet methods (filter/fields/get/delete/order_by etc.)
  • All QuerysetProxy methods (mostly same as with QuerySet, separately for many to many and reverse side of fk as they are handled a little different)
  • Inheritance:
    • Inheritance from Mixin and Concrete models
    • PKs declared in parent, and redeclared in children
    • FKs declared in parent and redeclared with children
    • PKs and FKs declared only in children
    • Operations on inherited models for all Model, QuerySet and QuerysetProxy
  • Self-relations and deferred resolving model with composite pk referencing itself
  • Deferred resolving of ForwardRefs in relations (fks and m2m)
  • Multiple relations between same two models
    • two fks between A and B,
    • two many to many between A and B,
    • one fk and one m2m between A and B
  • Optional and required FKs (now we didn't include a param for it, probably nullable: bool = True should be included in FK signature)
  • Columns of pk that are autoincrement/ have default/default function set and fields that are required.
  • Operations on one model alone
  • Operations on simple relation between A - B
  • Nested relations -> like B - C - D - E, with combination of fks and m2m
  • Relations with same model appearing multiple times, schema similar to:
        B - C - Z
      /
    A               
      \  
        D - C - Y
    
  • Explicitly created and implicitly created Through Models for m2m relation (Through Models also can have composite pk? Probably yes.)
  • Complex filters (and_/or_)
  • dict() and json() checks
  • Setting names for columns and relations:
    • aliases set for db columns in pks
    • aliases for db columns for fks
    • aliases for both pks and fks
    • default reverse relation names
    • setting different reverse relation names
    • non existing column names used in composite pks and fks (exceptions)
    • overlapping related names (i.e. same related_name from two different models) causes exception
  • Prohibiting encryption of pks and fks columns
  • Internals:
    • Composite fields has to be stored in Meta.model_fields dictionary -> probably changing ForeignKeyField and ManyToManyField should be enough, don't thing a new field type would be required, maybe something like is_composite flag on relation fields.
    • Since pks cannot be partial the pk property for composites should return None (not empty dict) if not set and dict with values if set
    • ormar.Integer fields that are part of composite pks should be set to autoincrement=True by default
    • pks are always included in the model, even if someone explicitly exclude_fields({pk_column(s)}) this call is "ignored" (pk column is added at the end regardless of anything else).
    • don't forget to black/flake8 and mypy the codebase -> for flake8 only ormar is important not the tests, for mypy/black both

Maybe something else will come to my mind. As you can see there is a vast amount of areas that needs to be checked to ensure proper working of ormar with this new feature. I can probably help you with writing those tests if you want me to ;)

@tillspeicher

Copy link
Copy Markdown
Author

Thanks for taking the time to go over the tests and creating the list of requirements!
Indeed, those are a lot of tests :) It feels to me like the list you mentioned covers a significant part of the general Ormar functionality. In that case, it might be easier to cover those cases by modifying the existing set of tests by changing or adding some models that have composite PKs and FKs.
For now, I want to start with the actual implementation, but I will look into modifying the tests to use composite PKs and FKs once that's done and you are fine with modifying the existing tests rather than writing additional ones. Of course, any help in doing that would be welcome :)

Mypy on the tests is producing errors for now, but that's expected since e.g. ForeignKeyConstraint is not implemented yet.
However, when I ran black as described in the contribution guidelines on the codebase, it reformatted a number of ormar and test files for me. As far as I can see those modifications were done to reduce the line length. Are you using a max-line-length that's different from the 88 characters default?

@collerek

Copy link
Copy Markdown
Collaborator

No, the lines length should be 88 if I remember correctly (that's also my pycharm and flake8 setting).

Yeah, that's the problem with pks -> they are used all over the place -> relations need to be resolved to them in joins, it's a default sort order if you do not provide additional one, they cannot be excluded in fields/exclude_fields etc. that's why a lot of tests are needed with this functionality to properly check all aspects on all db backends.

I'm ok with changing some of the tests (or preferably adding new models), but not all since then you loose tests for simple one column pks, and like they say never too much tests. I will help you with tests as that may be also an opportunity to clean them up a little, maybe divide into subpackages as it's starting to be a lot of files to grasp.

@collerek

collerek commented Jun 1, 2021

Copy link
Copy Markdown
Collaborator

@tillspeicher are you gonna continue this work or should I close this PR?

@tillspeicher

Copy link
Copy Markdown
Author

I'm sorry for going silent on the PR.
I got sidetracked with a different project than the one I need this for and had to prioritize that.
I have composite PKs implemented into the existing codebase with 15 tests still failing but haven't implemented an API for defining composite PKs and FKs yet, which will take some more time. I also need to merge this with the changes that happened to the codebase in the meantime.
I am planning on getting back to this issue, but can't say when yet. It's probably best to close the PR for now, once I get back to this project I can start a new one or reopen it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants