Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate header from yaml spec #259

Merged
merged 13 commits into from
Jan 22, 2024
Merged

Conversation

rajveermalviya
Copy link
Collaborator

@rajveermalviya rajveermalviya commented Jan 4, 2024

Add a custom yaml spec as discussed in #207, the details about the format can be found in the schema.json.

To test the generator:

  1. First make sure you have Go installed.
  2. Make some change in webgpu.yml
  3. Run make gen

The generator can also be used to generate a separate implementation specific C header along side webgpu.h. The steps to do that is:

  1. Create a new yaml file, for example wgpu.yml
  2. Populate it with minimal required fields, refer to schema.json and this example.
  3. Run go run ./gen -schema schema.json -yaml wgpu.yml -header wgpu.h

Another option is to generate a single header from multiple yaml specs, which is also included but disabled currently.
I prefer separate headers because that's what wgpu-native uses currently.

@rajveermalviya rajveermalviya force-pushed the yaml-gen branch 3 times, most recently from 9d9468c to e60158c Compare January 4, 2024 17:25
@rajveermalviya rajveermalviya marked this pull request as ready for review January 9, 2024 17:27
Copy link

@ben-clayton ben-clayton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go / template readability LGTM. Thank you for all the changes.
Review still needed from project maintainers.

@kainino0x
Copy link
Collaborator

Sorry for the delay, I've been having a hard time catching up with work. Thank you Ben for the Go readability review!

One high level comment - I'd like to have a blocking GitHub Actions check that the generated output is up-to-date. This shouldn't be too complicated, just run the generator and then check that the git repository is clean.

@kainino0x
Copy link
Collaborator

kainino0x commented Jan 20, 2024

(Oh, and - since we are generating files already - I wonder if we could also generate and check in a JSON version of the YAML file, for easy ingestion by other projects? But let's save this for a future PR.)

Copy link
Collaborator

@kainino0x kainino0x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have much to say about the Go code. I don't know Go, but it seems good, and if it works, it works :) Looking at the schema, it looks really great!

The changes to webgpu.h look really easy to reproduce in Dawn (except the callback definition order will require a little work), so we can stay in sync until we start using this generator in Dawn (hopefully).

I added some comments for future work, but let's probably do those in another PR.
Just a few little things I'd like to do before landing this, but they're not critical.

gen/main.go Outdated
// for _, o := range objects {
// yml.Objects = append(yml.Objects, o)
// }
// }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plan for this commented-out code?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed for now, plan is to add it back later (for finding invalid duplicates across multiple yaml specs).

"properties": {
"name": {
"type": "string",
"description": "Name of the constant variable/define"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Can be future work)
Maybe we could use regex patterns to validate these a little more?
https://json-schema.org/understanding-json-schema/reference/string#regexp

These would get peppered all over the place so we'd probably want to link to the definition by reference:
https://stackoverflow.com/a/62585173/1268131

schema.json Outdated
},
"value": {
"type": "string",
"description": "An unsigned integer in decimal or hexadecimal syntax, in string"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regex patterns are useful here too: ^(0x[0-9A-F]{4}|0|[1-9][0-9]{0,4})$ or something like that

Though perhaps better - since we're using YAML already which supports hexadecimal numbers - could we just accept integers here instead? With range validation:
https://json-schema.org/understanding-json-schema/reference/numeric#range
(Can also be future work)

schema.json Outdated
},
"optional": {
"type": "boolean",
"description": "Optional property, to indicate if a struct member is optional"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Can be future work)
For the INIT macros (and generating C++ bindings), some optional members will need to know what their default value is (like WGPU_DEPTH_SLICE_UNDEFINED). Probably we can just put that here, though an alternative could be to move that information into the "type system" and give the depthSlice member a special type that knows its default.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default values are planned - In-order to keep this initial PR minimal, a future PR which includes both the header generation and default values populated in the yaml at once would be better.

Copy link
Collaborator

@kainino0x kainino0x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Nice reduction on the schema, thanks for digging deeper into what we could do with ref.

@kainino0x
Copy link
Collaborator

I've left the future work comments "unresolved" as a reference.

@kainino0x
Copy link
Collaborator

I'll go ahead and merge since webgpu.h changes are minimal - in Dawn we can hopefully catch up soon so the sync process won't be made harder.

@kainino0x kainino0x merged commit 4f86313 into webgpu-native:main Jan 22, 2024
1 check passed
@rajveermalviya rajveermalviya deleted the yaml-gen branch January 23, 2024 03:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants