Overview of the Spring Integration Batch Module

Many use cases in Spring Batch look like they might be efficiently and concisely implemented in Spring Integration. Here is a list. These are features that can extend Spring Batch, or use Spring batch features in the context of Spring Integration. Work in progress waiting for community feedback. Many issues to do with transactionality and synchronous execution have been raised and fixed in Spring Integration as a result of these use cases being prototyped.

ID Description Status Sub-package Comments
1 Message triggers job Complete launch Complete. Also lots of opportunities with monitoring progress.
2 Chunking and multi-VM job execution Complete chunk Failures might need some analysis. Use of stateful StepExecutionListener requires use of step scope.
3 Asynchronous Aggregator Unstarted
4 Stateful and non-linear jobs -> job = flow Complete job Simple use cases work well with Spring Batch 2.0 and no Integration features.
5 Flexible item processing model (as message flow) -> step = flow Complete item Complete (v. simple using MessagingGateway). Unit tests only.
6 Automatic repeat / retry Complete retry (unit test) Unit tests only, since it just uses existing features.
7 Restartable file processing Complete file Seems to hang together. Not tested thoroughly, but apparently someone is using it.
8 Asynchronous item processing Complete async A general purpose ItemProcesor that returns a Future.

Numbers 2, 4, 5 have also been identified as high level Spring Batch 2.0 Features or themes. If we implement 1, then we also don't need to do any more scheduling and triggering in Spring Batch.

Number 6 from the list (repeat/retry) is more of a Spring Integration pattern than a Spring Batch one. We implemented it in Spring Batch first, with an eye to seeing about pushing it out into Spring Integration later (with probably a split of repeat/retry out of Batch at that time).

Message Triggers Job

Description:

  1. User sends message to channel (maybe through a scheduler)
  2. System interprets message payload as parameters for JobLauncher
  3. System launches job execution
  4. If message had a replyTo, System acknowledges with JobExecution
  5. User accepts response and uses it to monitor progress

Variation:

  1. System waits for job to finish and replies when it is over
  2. User polls for replies and gets notification about end of execution

Variation:

  1. User wants to block on send and only receive response when job is done

Chunking and Multi-VM

Description:

  1. Step flushes chunk as message to outgoing channel (repeat up to throttle limit)
  2. Worker thread picks up chunk and processes it
  3. Worker thread replies to response channel
  4. Step picks up reply and aggregates the counts
  5. Step blocks until all the requests are satisfied

TODO: failure modes

Asynchronous Aggregator

Job is executed over long period. Many jobs can be executing concurrently.

Description:

  1. Input stage for each job: System reads all items and marks with the job instance id in a durable repository (staging table)
  2. System sends each item (or chunks of items that can be processed together as appropriate) to a channel
  3. Items flow through message pipeline, occasionally pausing until certain conditions are met, possibly for days at a time
  4. Aggregator sits and waits for all items in a job to be finished and then wraps up

Stateful and non-linear jobs

Dependencies beyween steps and conditional flow between steps. Each handler node in a message flow is a step execution, with all the robustness guarantees from the Spring Batch meta data.

Description:

  1. User launches job
  2. System sends message to channel containing job execution
  3. Handler accepts message and executes a step
  4. Handler translates result of step execution into the same form that it accepted the original request
  5. System routes message to next handler, possibly dynamically based on data in the message
  6. Next handler does the same... until one of the routing decisions leads to a reply channel
  7. System receives reply and transfers information to job execution (e.g. status) as necessary

Variation: failure in one of the handlers

Variation: restart after failure

Flexible item processing model

Description:

  1. Step hands item to ItemWriter
  2. Item is converted to message and sent to synchronous flow
  3. Handler accepts message and does something with item
  4. System routes result to next handler, possibly dynamically

Variation: failure

  1. Handler throws exception
  2. System propagates exception up to ItemWriter (forces rollback under normal circs - hence synchronous flow)

Automatic repeat / retry

Description (repeat):

  1. User sends message to channel
  2. System start a transaction and reseives message, then processes it
  3. User sends another message
  4. System receives and processes it in the same transaction
  5. ... repeat ...
  6. System determines that batch is complete and commits transaction

Restartable file processing

Large files need to be processed, so message payload of file contents is not practical. One line or XML event per message with failover and restartability from Spring Batch.

Description:

  1. User triggers file processing (sends message, copies file to directory, etc.)
  2. System starts new job
  3. System processes file line by line (or even by event), wrapping each one as a message and sending it to a synchronous flow
  4. System commits periodically (as determined by Spring Batch step configuration)

Variation: failure and restart

  1. Item processing fails
  2. System aborts job and sends message to failure channel (or failure message to normal reply channel)
  3. Operator fixes problem and triggers restart (another message channel?)
  4. System restarts job for same file at point where it left off
  5. System completes processing
  6. System sends sucess message to reply channel

Variation: send to asynchronous flow. Same as main use case but item message is sent to asynchronous flow. Not as robust because if the lights go out then meesages will be lost, but at least a large file can be split into smaller chunks.

Asynchronous item processing

This is actually a variation on flexible item processing model.

Description (async):

  1. ItemProcessor executes in background (non-transactionally)
  2. ItemWriter collects outputs from futures before phyically writing data