Key Highlights​

• New option --connectors.max-receive-record-size in conduit run to configure the record size when using standalone connectors. More information.
• Processor split enables you to split an array in a record into separate records. This can be useful in AI pipelines where the input record contains a document and the output is multiple embeddings inserted into a database.
• Processor clone enables you to clone any record N times. This is useful when one input record needs to be split into multiple collections (tables) on the destination.
• Improvements for pipelines init command, which now creates a runnable demo pipeline when no flags are supplied.

Key Highlights​

• We've developed a destination connector for Box.
• The destination connectors support writing chunked files.
• The support for chunking comes from using standard OpenCDC metadata. That means that the Box destination connector can write large files read by other similar connectors (such as the Dropbox source connector).
• You can download the connector here.
• To learn how to use and configure the connector, please visit its documentation page.

Have questions or feedback? Join us on Discord.

Key Highlights​

• Add a validateConnection configuration parameter This release adds the new configuration option validateConnection, giving the user the freedom to choose whether to validate the connection to the HTTP server before streaming data or not. This can be useful if the source or destination's HTTP server don't support receiving HEAD requests (which is what the connector uses to validate the connection), or if the server is not expecting such a request. So now the user can choose if the connection will be validated or not.

Have questions or feedback? Join us on Discord.

Key Highlights​

• We've developed a source and a destination connector.
• The source and destination connectors support chunking of large files.
• Chunking is done using standard OpenCDC metadata. That means that the Dropbox can send large files to other similar connectors (such as the upcoming Box connector). It works the opposite way too!
• You can download the connector here.
• To learn how to use and configure the connector, please visit its documentation page.

Have questions or feedback? Join us on Discord.

Key Highlights​

• We refactored how we move logical replication messages in the connector and achieved a 14% boost in performance (verified through our benchmarks). Here's the relevant PR: #284.
• We also improved support for the NUMERIC/DECIMAL type in Postgres. So far, we were reading it only if it can fit into an int64 or float64. Now, the connector reads them as big.Rat values. For the curious ones, here's the PR: #290.

Have questions or feedback? Join us on Discord.

More information​

• Fix processor task handling filtered records (pipeline architecture v2) #2257
• New option --connectors.max-receive-record-size in conduit run to configure the record size when using standalone connectors. More information.
• Read symbolic links in the pipelines directory #2298
• The destination of Cohere processors is now configurable #2231
• Under certain circumstances, a pipeline with multiple connectors cannot be provisioned using a configuration file. We fixed that in #2314.

Key Highlights​

This update incorporates the latest Conduit Connector SDK, making it possible to add it as a built-in connector to your Conduit build using the latest Conduit version.

Have questions or feedback? Join us on Discord.

Key Highlights​

• DynamoDB Destination Connector Added This release brings a destination connector to complement the existing DynamoDB source. It supports all standard operations: snapshot, insert, update, and delete, making it easier to stream data into DynamoDB tables from other sources.

Have questions or feedback? Join us on Discord.

If you're using InfluxDB, you're ready to use it with Conduit as a standalone connector, or as part of a custom Conduit Build.

More information​

• Add a new Ollama Processor as built-in processor.
• Add the ability to easily configure Processor Plugins when building a custom Conduit. Check out the documentation page for more information. Thanks @nickchomey for the contribution!
• New configuration option preview.pipeline-arch-v2-disable-metrics to disable metrics for the new pipeline architecture.
• Fixes a bug where the ENV variable CONDUIT_CONFIG_PATH didn't seem to work propertly.
• Fixes a bug when using the default processor middleware.

Key Highlights​

• Add settings for skipping queue and exchange declare call. You can now configure the connector to skip declaring queues and exchanges, providing more flexibility for advanced RabbitMQ setups by @rauanmayemir.
• Implement publisher confirmations. Added support for publisher confirmations, improving message delivery reliability by @alarbada.

If you have any questions, or more feedback feel free to join our Discord Community.

Key Highlights​

• Make AWS credentials optional with fallback to default credential chain. This change allows the connector to authenticate with AWS using either static credentials or AWS default config chain, providing more flexibility in how the connector accesses DynamoDB resources. Thanks @annsway for the contribution!

If you have any questions, or more feedback feel free to join our Discord Community.

Key Highlights​

• Record Batch Support and Multi-Collection Mode. Added support for processing records in batches and introduced multi-collection mode for better flexibility by @alarbada.
• New snapshot.enabled Source Configuration Parameter. Introduced a snapshot.enabled parameter to allow users to skip the initial snapshot phase when configuring the source by @alarbada. (note: this was initially introduced by #131).
• Composite Key Support. Added support for composite keys, enabling more complex primary key configurations by @alarbada.
• Regex Support for Canal CDC Iterator. Enhanced the Canal CDC iterator to accept regex patterns for more granular control over data processing by @nickchomey.
• Improved Schema Generation. Fixed an issue where certain MySQL types containing integers in parentheses were not handled correctly during schema generation by @derElektrobesen.
• Bug Fix: CDC Iterator Record Loss. Fixed a bug where some records could be lost during CDC iterator reads by @derElektrobesen.

Thanks to the contributions from our community, we are able to continuously improve the Conduit MySQL Connector. We appreciate your feedback and support!

If you have any questions, or more feedback feel free to join our Discord Community.

Key Highlights​

• Destination-only connector
• Create + Stream operations supported
• Target specific folders: Control where files are stored in Drive.

If you have any questions, or more feedback feel free to join our Discord Community

More information​

New source field aws.sessionToken to connect to the source connector with a session token.

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      - id: example
        plugin: "dynamodb"
        settings:
          # AWS temporary session token. Note that to keep the connector running
          # long-term, you should use an IAM user with no temporary session
          # token. If the session token is used, then the connector will fail
          # once it expires.
          # Type: string
          # Required: no
          aws.sessionToken: ""

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      - id: example
        plugin: "dynamodb"
        settings:
          # AWS access key id.
          # Type: string
          # Required: yes
          aws.accessKeyId: ""
          # AWS secret access key.
          # Type: string
          # Required: yes
          aws.secretAccessKey: ""

If you have any questions, or more feedback feel free to join our Discord Community

More information​

ReadN addition​

Since the introduction of end-to-end batching on the Connector SDK, (part of the Connector SDK 0.12.0), source connectors could add support for batching by implementing the ReadN method.

Validating logical replication slot name​

This validation was introduced by one of our newest Conduit users. Thanks Rauan Mayemir!

Rauan reported that publications are fine with special characters in the name, but not replication slots, so with Conduit Connector Postgres 0.12.0, you can ensure that the replication slot name is valid before starting the connector.

If you have any questions, or more feedback feel free to join our Discord Community!

Changes​

Docker Image Update​

The RabbitMQ Docker image has been updated to version ⁠4.1, ensuring compatibility with the latest RabbitMQ features and improvements.

Metadata Propagation to RabbitMQ Headers​

Record metadata keys prefixed with rabbitmq.header. are now propagated to AMQP 0.9 headers (and vice versa).

This feature allows users to pass custom headers when publishing messages to RabbitMQ.

Example:

version: 2.2
pipelines:
  - id: rabbitmq-headers-example
    connectors:
      - id: source
        # other connector configuration
        processors:
          - id: add-headers
            plugin: builtin:field.set
            settings:
              field: metadata.rabbitmq.header.app_id
              value: "inventory-service"
      destination:
        plugin: rabbitmq
        config:
          ...

In this example, the messages sent to RabbitMQ will include app_id in the headers:

{
  "app_id": "inventory-service",
}

RoutingKey as a Go Template​

The routingKey configuration now supports Go templates, enabling dynamic routing key generation based on metadata.

• Breaking Change: This replaces the previous static routingKey implementation.
• Improvement: Users can now use a single pipeline with a processor to set rabbitmq.routingKey dynamically, reducing the need for multiple pipelines.

Example: A processor can set rabbitmq.routingKey to a desired value, which will then be used as the topic exchange routing key.

Thanks @rauanmayemir for these contributions!

If you have any questions, or more feedback feel free to join our Discord Community

More information​

Thanks to the report by one of our community members, we identified a bug in ecdysis that caused the environment variable CONDUIT_CONFIG_PATH not to be respected.

This environment variable is used to specify the path to the configuration file for Conduit, and it's one way to set the configuration for Conduit (check out the docs for more information).

This was added to Conduit on this pull-request, and will be part of the next Conduit release.

conduit run --config.path /path/to/conduit.yaml

If you have any questions, or more feedback feel free to join our Discord Community!

More information​

v0.13.4 comes with two fixes for configuration parameters that contain a dash (-) in their names. The fixes are:

• Bug: Certain environment variables are not respected
• Fix configuration for the Postgres connection string

Thanks to one of our community members that let us know about the Postgres connection string bug!

It was a long-awaited feature, and we're happy to announce that it's finally possible to read our documentation at night without straining your eyes 🎉.

The theme will automatically change based on your system preferences, but you can also manually switch between light and dark themes using the toggle in the top right corner of the site.