Concepts: Composition Instructions¶

Dracon provides special instruction tags (!define, !if, !each) that allow you to embed logic directly into your YAML. These instructions operate during the composition phase, manipulating the YAML node tree and context before includes, merges, or final Python object construction occurs. There are also construction-phase tags like !noconstruct that control what gets built into Python objects.

Defining Variables (`!define`, `!set_default`)¶

These instructions create variables within the configuration's context, making them available for subsequent interpolation (${...}, $(...)) or use within other instructions (!if, !each).

!define var_name: node_value
Assigns the resulting value to var_name in the context of the current node and its descendants.
Overwrites any existing variable with the same name in the current scope.
The !define var_name: ... entry is removed from the final configuration structure.
For expressions (${...}): evaluated immediately at composition time.
For typed objects (!MyModel { ... }): construction is deferred until the variable is first accessed via ${var_name}. This means forward references work -- the object can reference variables defined later in the file.
!set_default var_name: node_value
Similar to !define, but only sets var_name if it does not already exist in the current context scope.
Useful for providing defaults that can be overridden by parent contexts or earlier includes.
Also removed from the final configuration structure.

# --- Example ---
!define app_version: "1.2.0" # Simple string definition
!define is_prod: ${getenv('ENV') == 'production'} # Evaluated now using env var
!set_default log_level: "INFO" # Set only if not already defined

config:
  version: ${app_version} # Uses "1.2.0" (available from parent scope)
  debug_mode: ${not is_prod} # Uses the boolean calculated earlier
  logging:
    level: ${log_level} # Uses "INFO" unless overridden elsewhere

Final config object: { "version": "1.2.0", "debug_mode": ..., "logging": { "level": "INFO" } } (The !define keys are gone).

Lazy Construction with Typed Objects¶

When !define assigns a typed node (a mapping or sequence with a !TypeTag), construction is lazy -- it happens when the variable is first accessed, not when the !define is processed. This is important because it means the object's interpolations are resolved in the full context, including variables defined after the !define:

# order doesn't matter -- 'model' is constructed when ${model} is accessed
!define model: !Predictor
  data: ${dataset}
  method: linear
!define dataset: !DataLoader { path: data.csv }

# both variables are available here
output: ${model.predict()}

The result is the real Python object. isinstance() returns True, methods work, attributes are accessible. If the variable is never referenced, the object is never constructed.

This replaces the old !noconstruct + construct(&/name) ceremony entirely.

Explicit Type Coercion (`!define:type`)¶

YAML's implicit type inference usually does the right thing (1.2 is a float, 42 is an int). But sometimes you need a different type than what YAML infers. The !define:type syntax applies explicit coercion:

!define:float one: 1          # float 1.0, not int 1
!define:str port: 8080         # string "8080", not int
!define:bool enabled: 1        # True, not int 1
!define:int threshold: 3.7     # int 3, not float

Supported types: int, float, str, bool, list, dict. This syntax also works with !define?:type and !set_default:type.

YAML Functions (`!fn`)¶

!fn wraps a YAML template -- file or inline -- into a first-class callable. It bridges composition-time structure and expression-time computation: an !include is a statement (merges into scope), but !fn produces a value (composable in expressions).

!define make_endpoint: !fn
  !require name: "service name"
  !set_default port: 8080
  url: https://${name}.example.com:${port}

# tag syntax -- looks like a type constructor
api: !make_endpoint { name: api, port: 443 }

# expression syntax -- composable
all: ${[make_endpoint(name=n) for n in service_names]}

From the caller's perspective, a YAML callable and a Python class are the same thing. Both are tags that take keyword arguments and produce structured results. The implementation (YAML template vs Python class) is invisible.

Each invocation gets a fresh scope. The template is deep-copied, arguments are injected as context variables, and the full composition + construction pipeline runs. !require validates that mandatory arguments are provided; !set_default provides fallbacks. Arguments are consumed by these instructions and don't appear in the output.

This fills the gap between one-off expressions and full file includes:

Mechanism	What it is	Scope
`${expr}`	Inline expression	Single value
`!define x: val`	Variable binding	Current scope + descendants
`!fn`	YAML-defined function	Reusable, isolated, composable
`!include`	File composition	Structural (merges into tree)
`!Type { ... }`	Python constructor	Type-checked object

See the YAML Functions guide for full usage patterns.

Partial Application (`!fn:path`)¶

Where !fn wraps a YAML template into a callable, !fn:path wraps an actual Python function. The colon suffix is a dotted import path (or context name) that resolves to a callable. The mapping body provides pre-filled kwargs that are stored and merged with runtime kwargs on each call.

loss_fn: !fn:biocomp.train.energy_loss
  kl_weight: !optax.polynomial_schedule { init_value: 0.1, end_value: 0.001 }
  energy_weight: 0.5

The result is a DraconPartial -- calling loss_fn(stack, config) invokes energy_loss(stack, config, kl_weight=<schedule>, energy_weight=0.5). Runtime kwargs override stored ones.

This is functools.partial with two extras: YAML round-trip serialization (dracon.dump produces !fn:path { kwargs }) and pickle support (the function is stored as its import path string).

With no mapping body, !fn:path is a serializable function reference:

activation: !fn:jax.nn.relu

The path resolves in context first, then as a dotted import path. Nested tags and interpolations in the kwargs are resolved at construction time, so !fn:f { x: !g { y: 1 } } calls g(y=1) once and stores the result as x.

	`!fn` (template)	`!fn:path` (partial)
Wraps	YAML template	Python function
Scope	Composition (YAML manipulation)	Construction (Python call)
Kwargs resolved	Each call re-composes	Once at construction
Serializable	No	Yes (pickle + YAML dump)
Use case	Config templates, YAML generation	Runtime callables, loss fns, optimizers

Function Composition (`!pipe`)¶

Where !fn turns a template into a callable, !pipe composes callables into a pipeline. The result is itself a callable -- usable as a tag, in expressions, or as a stage in another pipe.

!define load: !fn file:templates/load.yaml
!define clean: !fn file:templates/clean.yaml
!define train: !fn file:templates/train.yaml

!define ml: !pipe [load, clean, train]
result: ${ml(path='/data/train.csv', model_type='xgb')}

Each stage's output feeds into the next. If a stage returns a mapping, its keys are unpacked as keyword arguments to the next stage. If it returns a typed object (like a Pydantic model), it goes as a single value to the next stage's unfilled !require parameter. This means stages written for standalone use automatically work inside a pipe.

Pre-fill kwargs per stage with mapping syntax:

!define ml: !pipe
  - load
  - clean: { strategy: aggressive }
  - train

Pipeline kwargs flow through to all stages. Pipes compose with other pipes (nested pipes are flattened).

!fn:path nodes can be used as inline pipe stages -- any tagged node in the sequence is constructed and validated as callable:

!define pipeline: !pipe
  - !fn:preprocess.load_data
  - !fn:preprocess.clean { strategy: aggressive }
  - !fn:models.train

This mixes freely with named stages and !fn templates.

Primitive	What it does
`!define`	Name a value
`!fn`	Define a function (template to callable)
`!fn:path`	Partial-apply a Python function (serializable)
`!pipe`	Compose functions (callables to callable)
`!if`	Branch
`!each`	Iterate
`!require` / `!set_default`	Declare parameters

Variable Contracts (`!require`)¶

While !define and !set_default provide values, !require declares that a variable must be provided by an outer scope -- a parent file, cascade overlay, CLI ++var=value, or another !define. If nobody provides it by end of composition, a clear error is raised.

# base.yaml -- shipped with the tool
!require environment: "set via ++environment or create a .myapp.yaml overlay"
!require api_key: "set API_KEY env var or provide in overlay"

endpoint: https://${environment}.api.example.com
auth:
  key: ${api_key}

This completes the variable definition gradient:

!define -- always set, overwrites previous values
!set_default -- set if nobody else does (optional with fallback)
!require -- must be provided by someone else (mandatory, no fallback)

Composition-Time Assertions (`!assert`)¶

!assert validates arbitrary invariants over the composed tree. The expression uses the same interpolation engine as ${...}. Assertions run after all other instructions are resolved, so they validate the final composed state.

!assert ${port > 0 and port < 65536}: "port out of range"
!assert ${not (environment == 'prod' and debug)}: "debug must be off in prod"

Assertions are removed from the final tree -- pure validation, zero runtime overhead.

Conditional Composition (`!if`)¶

Includes or excludes configuration blocks based on a condition evaluated at composition time.

Syntax: !if <condition_expr>: <node_value> (must be a mapping key)
Behavior:
<condition_expr> is evaluated. It can be a boolean literal (true/false), an integer (0 is false, others true), a string (non-empty is true), or an interpolation (${...}) resolving to a truthy/falsy value at composition time.
Shorthand format (no then/else keys in value):
- If True: The value's key-value pairs are merged into the parent mapping.
- If False: The !if entry is removed entirely.
then/else format (value mapping contains then and/or else keys):
- If True: The then branch's content is merged into the parent mapping.
- If False: The else branch's content is merged (or nothing if no else).

!define enable_feature_x: ${getenv('FEATURE_X') == 'true'}
!define env: "prod"

settings:
  base_setting: true
  # This block included only if enable_feature_x is true
  !if ${enable_feature_x}:
    feature_x_url: "http://feature-x.svc"
    retries: 5

  # This block included because env == "prod" evaluates to true
  !if ${env == "prod"}:
    monitoring: full
    sampling: 0.1

  # This block removed because env == "dev" is false
  !if ${env == "dev"}:
    debug_endpoint: "/_debug"

Iterative Composition (`!each`)¶

Generates multiple configuration nodes by iterating over a list or other iterable evaluated at composition time.

Syntax: !each(<loop_var>) <iterable_expr>: <node_template>
Behavior:
<iterable_expr> (e.g., ${list_variable}, ${range(3)}) is evaluated at composition time to produce an iterable.
For each item in the iterable:
- A deep copy of <node_template> is made.
- A temporary context { <loop_var>: item } is merged into the copied node's context (overriding any existing <loop_var>).
- If <node_template> is a sequence item (- value), the processed copy is appended to the resulting list.
- If <node_template> is a mapping (key: value), the processed copy's key-value pairs are added to the resulting dictionary. Keys within the mapping template often must use interpolation (e.g., key_${loop_var}) to ensure uniqueness.
The original !each... entry is replaced by the generated list or dictionary.

!define user_list: ["alice", "bob"]
!define service_ports: { web: 80, api: 8080 }

config:
  # Generate a list of user objects
  users:
    !each(name) ${user_list}: # Iterate over the list variable
      - user_id: ${name.upper()} # Use loop_var 'name'
        home: "/home/${name}"

  # Generate a dictionary of service configs
  services:
    ? !each(svc_name) ${service_ports.keys()} # Iterate over dict keys
      # Use loop_var in the key for uniqueness
    : ${svc_name}_config:
        port: ${service_ports[svc_name]} # Access original dict using loop_var
        protocol: http

Resulting config (before final construction):

config:
  users:
    - user_id: ALICE
      home: "/home/alice"
    - user_id: BOB
      home: "/home/bob"
  services:
    web_config:
      port: 80
      protocol: http
    api_config:
      port: 8080
      protocol: http

Excluding Nodes (`!noconstruct`, `dracon`)¶

Sometimes you need helper nodes or templates during composition that shouldn't appear in the final constructed configuration object.

You probably don't need !noconstruct for building objects

The common pattern of !noconstruct + construct(&/ref) to manually build Python objects from YAML nodes has been replaced by lazy !define. Just write !define x: !MyType { ... } and use ${x} -- construction happens automatically on first access. See the lazy construction section above.

!noconstruct <node>
Applies to any node (scalar, sequence, mapping).
The node exists during composition (can be referenced via anchors &, includes !include, or defines !define) but it and its children are skipped during the construction phase -- they won't appear in the final Python object.
Still useful for template anchors, metadata, and nodes that should exist in the tree but not in the output.
__dracon__<key>: ...
Applies only to top-level keys in a mapping.
Any key starting with __dracon__ behaves as if !noconstruct was applied to its value.
Provides a convenient namespace for composition-only helpers.

# Define a template but hide it from the final output
!noconstruct &service_defaults:
  timeout: 60
  protocol: https

# Alternative using __dracon__ namespace for organization
__dracon__templates:
  db_defaults: &db_defaults
    pool_size: 10
    encoding: utf8

# --- Use the templates ---
http_service:
  <<: *service_defaults # Include the copy (deep copy)
  protocol: http        # Override protocol

database:
  <<: *db_defaults     # Include the copy

Final constructed config:

{
  "http_service": {"timeout": 60, "protocol": "http"},
  "database": {"pool_size": 10, "encoding": "utf8"}
}
# The '!noconstruct' node and '__dracon__templates' key are gone.

Concepts: Composition Instructions¶

Defining Variables (!define, !set_default)¶

Lazy Construction with Typed Objects¶

Explicit Type Coercion (!define:type)¶

YAML Functions (!fn)¶

Partial Application (!fn:path)¶

Function Composition (!pipe)¶

Variable Contracts (!require)¶

Composition-Time Assertions (!assert)¶

Conditional Composition (!if)¶

Iterative Composition (!each)¶

Excluding Nodes (!noconstruct, __dracon__)¶