Adding DataSense Support

Static metadata is metadata that is known at compile time and available from the types in the connector’s JAR file. Simple Java types are part of this static metadata as are custom POJOs defined by the developer in its model. What is relevant is that the structure is well known or can be introspected without depending on other parameters.

Dynamic metadata is metadata of a type with an unknown structure at compile time. It has to be resolved based on the configuration of the connector at design time. This metadata has different uses that span a range of cases. Uses span from cases where the type itself is known but its structure depends on the service configuration (for example, where the system has an account with fields that can be customized by the user, so its structure has to be discovered each time based on the user’s credentials) to cases where everything is dynamic and the structure described heavily depends on the configured parameters (for example, the described structure is the payload of a service operation that also depends on the definition of the service based on the user sandbox).

To resolve dynamic metadata, the element has to be associated with a metadata resolver that knows how to obtain the desired type structure based on the current component configuration. This is explained in more detail in MetaData Resolvers.

Input metadata is the type resolution for the parameters of a component. Each parameter can provide either static or dynamic metadata that is isolated from the kind of metadata that other parameters of the same component expose.

Only operations and sources can have parameters with dynamic metadata, while configurations and connections always have static metadata.

The output of a component is bound to a static or dynamic resolution of its metadata. For example, a void operation has static metadata indicating that the output of that operation is of VoidType, while an operation that fetches a User profile from a remote service can dynamically describe the structure of the User type.

As explained in Structure, an operation can output a result that contains data for the payload and the attributes of the resulting message. The structure of these attributes is just as important as the structure of the payload. So, to improve the experience of your end user, metadata for each of them can be described independently, but dynamic metadata for attributes is always optional because some components might not produce attributes.

Note that certain return types are forced to describe their metadata dynamically. This requirement is for consistency because it is necessary to consider the experience of the end user, and types that are too generic interfere with the quality of the user experience.

To describe a dynamic metadata structure, you need to know what type has to be represented. This type reference is done by defining a @MetadataKeyId parameter in the operation that will hold the ID of the type to be described.

This means, for example, that if your operation can save a generic record to Amazon S3, but you want to provide a better design-time experience by describing each supported type structure (for example, Account or Organization), then one of your operation parameters will be a type reference containing either the id Account or the id Organization. This parameter will be used to describe either the Account structure or the Organization structure for the record to be saved into the bucket, depending on what the end user decides to use.

Having a @MetadataKeyId parameter is not always required because you might have only one structure (and not multiple) to define that vary dynamically based on the configuration. For example, you might have a User entity whose structure changes based on whether the connection credentials are for an administrator or not.

The purpose of including a MetadataKey parameter is to use the value given to it to resolve the metadata of output and inputs of the component for which the key is defined. This means that you should not define a MetadataKey without any MetadataResolver for inputs, output, or Attributes. SDK versions 1.3 and later fail to compile connectors that contain a Component that has a MetadataKey parameter but does not have an Output Resolver or Input Resolvers.

Whenever you want to obtain a dynamic element for DataSense, you implement a metadata resolver. There are many different kinds of resolvers, each with its own responsibility and use cases (explained later), but they all share these main concepts: