Fineract Platform Documentation

A Fineract instance is ONLY able to serve read API calls when it’s configured as a read or write instance. In batch instance mode, it won’t serve read API calls.
If it’s a read or write instance, the read APIs will be served.
If it’s a batch instance, the read APIs won’t be served and a proper HTTP status code will be returned.
The distinction whether something is a read API can be decided based on the HTTP request method. If it’s a GET, we can assume it’s a read call.

A Fineract instance is ONLY able to serve write API calls when it’s configured as a write instance. In read or batch instance mode, it won’t serve write API calls.
If the write APIs won’t be served and a proper HTTP status code will be returned.
If it’s a write instance, the write APIs will be served except the ones related to batch jobs.
The distinction whether something is a write API can be decided based on the HTTP request method. If it’s non-GET, we can assume it’s a write call. Also, the write APIs related to batch jobs (starting/stopping jobs) will not be served either.

A Fineract instance is ONLY able to serve batch API calls when it’s configured as a batch instance. In read or write instance mode, it won’t serve batch API calls.
If the batch APIs won’t be served and and a proper HTTP status code will be returned.
If it’s a batch instance, the batch APIs will be served.

Batch jobs are scheduled only if the Fineract instance running as a batch instance

When a Fineract instance is running in read-only mode, all event receiving/sending will be disabled.

With read separation, there’s a possibility to use read-only database connections for read-only instances.
If the instance is read-only , the DataSource connection used for the tenant will be read-only.
If the instance is read-only and the configuration for the read-only datasource is not set, the application startup will fail.

When a Fineract instance is running as batch, event receiving will be disabled while sending events will be still possible since the batch jobs are potentially generating business events.

After dropping the flyway migrations table (schema_version), Liquibase runs it’s
own migrations which fails (in recreating tables which already exisit) because
we are aiming to re-use DB with existing data from Fineract 1.5.0.

Solution: The latest release version (1.6.0) doesn’t have Liquibase at all, it
still runs Flyway migrations. Only the develop branch (later to be 1.7.0) got
sqitched to Liquibase. Do not pull the develop before upgrading your instance.

Make sure first you upgrade your instance (aka database schema with Fineract 1.6.0).
Then upgrade with the current develop branch. Check if some migration scripts
did not run which led to some operations failing due to slight differences in
schema. Try with running the missing migrations manually.

Note: develop is considered unstable until released.

Solution: A database upgrade is separate thing to take care of.

1.6 version shouldn’t include Liquibase. It will only be released after 1.6.
Make sure Liquibase is dropping schema_version table, as there is no Flyway
it is not required. Drop Flyway and use Liquibase for both migrations and
database independence. In case, if you still get errors, you can use git SHA
746c589a6e809b33d68c0596930fcaa7338d5270 and Flyway migration will be done to
the latest.

The jobs in Fineract can be divided into 2 categories:

Most of the jobs are normal batch jobs with limited scalability because Fineract is still passing through the evolution on making most of them capable to process a high-volume of data.

State management for the batch jobs is done by the Spring Batch provided state management. The data model consists of the following database structure:

The corresponding database migration scripts are shipped with the Spring Batch core module under the org.springframework.batch.core package. They are only available as native scripts and are named as schema-.sql where is the short name of the database platform. For MySQL it’s called schema-mysql.sql and for PostgreSQL it’s called schema-postgresql.sql.
When Fineract is started, the database dependent schema SQL script will be picked up according to the datasource configurations.

Chunking data has not been easier. Spring Batch does a really good job at providing this capability.

In order to save resources when starting/committing/rollbacking transactions for every single processed item, chunking shall be used. That way, it’s possible to mark the transaction boundaries for a single processed chunk instead of a single item processing. The image below describes the flow with a very simplistic example.

In addition to not opening a lot of transactions, the processing could also benefit from JDBC batching. The last step - writing the result into the database - collects all the processed items and then writes it to the database; both for MySQL and PostgreSQL (the databases supported by Fineract) are capable of grouping multiple DML (INSERT/UPDATE/DELETE) statements and sending them in one round-trip, optimizing the data being sent over the network and granting the possibility to the underlying database engine to enhance the processing.

Spring Batch provides a really nice way to do remote partitioning. The 2 type of objects in this setup is a manager node - who splits and distributes the work - and a number of worker nodes - who picks up the work.

In remote partitioning, the worker instances are receiving the work via a messaging system as soon as the manager splits up the work into smaller pieces.

Remote partitioning could be done 2 ways in terms of keeping the job state up-to-date. The main difference between the two is how the manager is notified about partition completions.

One way is that they share the same database. When the worker does something to a partition - for example picks it up for processing - it updates the state of that partition in the database. In the meantime, the manager regularly polls the database until all partitions are processed. This is visualized in the below diagram.

An alternative approach to this - when the database is not intended to be shared between manager and workers - is to use a messaging system (could be the same as for distributing the work) and the workers could send back a message to the manager instance, therefore notifying it about failure/completion. Then the manager can simply keep the database state up-to-date.

Even though the alternative solution decouples the workers even better, we thought it’s not necessary to add the complexity of handling reply message channel to the manager.

Also, please note that the partitioned job execution is multitenant meaning that the workers will receive which tenant it should do the processing for.

Business steps are a smaller unit of work than regular Spring Batch Steps and the two are not meant to be mixed up because there’s a large difference between them.

A Spring Batch Step’s main purpose is to decompose a bigger work into smaller ones and making sure that these smaller Steps are properly handled within a single database transaction.

In case of a business step, it’s a smaller unit of work. Business steps live within a Spring Batch Step. Fundamentally, they are simple classes that are implementing an interface with a single method that contains the business logic.

Here’s a very simple example:

The business steps are configurable for certain jobs. The reason for that is because we want to allow the possibility for Fineract users to configure their very own business logic for generic jobs, like the Loan Close Of Business job where we want to do a formal "closing" of the loans at the end of the day.

All countries are different with a different set of regulations. However in terms of behavior, there’s no all size fits all for loan closing.

For example in the United States of America, you might need the following logic for a day closing:

While in Germany it should be:

These are just examples, but you get the idea.

The business steps are configurable through APIs:

Retrieving the configuration for a job:

Updating the business step configuration for a job:

The business step configuration for jobs are tracked within the database in the m_batch_business_steps table.

By introducing the business day concept we are not tied anymore to the physical calendar of the system or the tenant. We got the ability to define our own business day boundaries which might end 15 minutes before midnight and any incoming transactions after the cutoff will be accounted for the following business day.

It is a logical date which makes it possible to separate the business day from the physical calendar:

Closing a Business Day could be a longer process (see COB jobs) meanwhile some processes shall still be able to create transactions for that business day (COB jobs), but others are meant to create the transactions for the next (incoming transactions): Business date concept is there to sort that out.

Business date concept is essential when:

This logical date is manageable via:

To maintain such separation from physical calendar we need to introduce the following new dates:

The - logical - date of the actual business day, eg: 2022-05-06

The - logical - date of the business day for job execution, eg: 2022-05-05

The event framework must support ACID guarantees on the business operation level.

Let’s see a simple use-case:

What happens if step 3 fails? Shall it fail the original loan creation process?

What happens if step 2 fails but step 3 still gets executed? We’re raising an event for a loan that hasn’t been created in reality.

Therefore, raising an event is tied to the original business transaction to ensure the data that’s getting written into the database along with the respective events are saved in an all-or-nothing fashion.

The system is able to send the raised events to downstream message channels. The current implementation supports the following message channels:

The events that are raised will be sent to the downstream message channels in the same order as they were raised.

The framework supports the at-least-once delivery guarantee for the raised events.

In terms of reliability and fault-tolerance, the event framework is able to handle the cases when the downstream message channel is not able to accept events. As soon as the message channel is back to operational, the events will be sent again.

Whether or not an event must be sent to downstream message channels for a particular Fineract instance is configurable through the UI and API.

All the events sent to downstream message channels are conforming a standardized format using Avro schemas.

The event framework is capable of being easily extended with new events for additional business operations or customizing existing events.

The event framework makes it possible to sort of queue events until they are ready to be sent and send them as a single message instead of sending each event as a separate, individual one.

For example during the COB process, there might be events raised in separate business steps which needs to be sent out but they only need to be sent out at the end of the COB execution process instead of one-by-one.

On a high-level, the concept looks the following. An event gets raised in a business operation. The event data gets saved to the database - to ensure ACID guarantees. An asynchronous process takes the saved events from the database and puts them onto a message channel.

The flow can be seen in the following diagram:

The whole framework is built upon an existing infrastructure in Fineract; the Business Events.

As a quick recap, Business Events are Fineract events that can be raised at any place in a business operation using the BusinessEventNotifierService. Callbacks can be registered when a certain type of Business Event is raised and other business operations can be done. For example when a Loan gets disbursed, there’s an interested party doing the Loan Arrears Aging recalculation using the Business Event communication.

The nice thing about the Business Events is that they are tied to the original transaction which means if any of the processing on the subscriber’s side fail, the entire original transaction will be rolled back. This was one of the requirements for the Reliable event framework.

The database plays a crucial part in the framework since to ensure transactionality, - without doing proper transaction synchronization between different message channels and the database - the framework is going to save all the raised events into the same relational database that Fineract is using.

For serializing events, Fineract is using Apache Avro. There are 2 reasons for that:

There are 3 different levels of Avro schemas used in Fineract for the Reliable event framework which are described below.

One of the requirements for the framework is to provide ordering guarantees. All the events have to conform a happens-before relation.

For the downstream consumers, this can be verified by the id attribute within the messages. Since it’s going to be a strictly-monotonic numeric sequence, it can be used for ordering purposes.

For easier consumption, the terminology event category is introduced. This is nothing else but the bounded context an event is related to.

For example the LoanApprovedBusinessEvent and the LoanWaiveInterestBusinessEvent are both related to the Loan bounded contexts.

The category in which an event resides in is included in the message under the category attribute.

The existing event categories can be found under the Event categories section.

The events stored in the database will be picked up and sent by a regularly executed job.

This job is a Fineract job, scheduled to run for every minute and will pick a number of events in order. Those events will be put onto the downstream message channel in the same order as they were raised.

The events database table is going to grow continuously. That’s why Fineract has a purging functionality in place that’s gonna delete old and already sent events.

It’s implemented as a Fineract job and is disabled by default. It’s called TBD.

Raising events is really easy. An instance of a BusinessEvent interface is needed, that’s going to be the event. There are plenty of them available already in the Fineract codebase.

And that’s pretty much it. Everything else is taken care of in terms of event data persisting and later on putting it onto a message channel.

An example of event raising:

Raising bulk events is really easy as well. The 2 key methods are:

First, you have to start recording your events. This recording will be applied for the current thread. And then you can raise as many events as you want with the regular BusinessEventNotifierService#notifyPostBusinessEvent method, but they won’t get saved to the database immediately. They’ll get "recorded" into an internal buffer.

When you stop recording using the method above, all the recorded events will be saved as a bulk event to the database; and serialized appropriately.

From then on, the bulk event works just like any of the event. It’ll be picked up by the processor to send it to a message channel.

TBD

TBD

Creating new events is super easy. Just create an implementation of the BusinessEvent interface and that’s it.

From then on, you can raise those events in the system, although you can’t publish them to an external message channel. If you have the event framework enabled, it’s going to fail with not finding the appropriate serializer for your business event.

The interface looks the following:

Quite simple. The get method should return the data you want to pass within the event instance. The getType method returns the name of the business event that’s gonna be saved as the type into the database.

First let’s talk about the event serializers because that’s what’s needed to make a new event compatible with the framework.

The serializer has a special interface, BusinessEventSerializer.

An implementation of this interface shall be registered as a Spring bean, and it’ll be picked up automatically by the framework.

New Avro schemas can be easily created. Just create a new Avro schema file in the fineract-avro-schemas project under the respective bounded context folder, and it will be picked up automatically by the code generator.

Apache Avro by default doesn’t support complex types like a BigDecimal. It has to be implemented using a custom snippet like this:

It’s a 20 precision and 8 scale BigDecimal.

Obviously it’s quite challenging to copy-paste this snippet to every single BigDecimal field, so there’s a customization in place for Fineract.
The type bigdecimal is supported natively, and you’re free to use it like this:

In case there’s a need some extra bit of information within the event message that the default serializers are not providing, you can override this behavior by registering a brand-new custom serializer (as shown above).

Since there’s a priority order of serializers, the only thing the custom serializer need to do is to be annotated by the @Order annotation or to implement the Ordered interface.

An example custom serializer with priority looks the following:

TBD

TBD

The rules to replace the Note services are very simple. If you provide an alternative implementation of the services then the default implementations will not be loaded.

Lookup value that is used to pick a loan transaction processor (see processor factory).

Descriptive text about the loan transaction processor that is mostly used in user interfaces.

TBD

TBD

TBD

TBD

TBD

TBD

TBD

The Gherkin language has the following keywords:

There are a couple of additional signs used in Gherkin:

Each non-empty line of a test specification needs to start with one of these keywords. The text blocks that follows the keywords are mapped to so called step definitions that contain the actual test code.

A typical Cucumber test specification written in Gherkin looks like this:

The corresponding step definitions would look like this:

As a proof of concept we’ve converted all unit tests in fineract-provider into Cucumber tests. The more interesting part starts when we’ll attack the integration tests with over 400 mostly business logic related tests. These tests fit very well in Cucumber’s test specification structure (a lot of if-then-else or in Gherkin: Given-When-Then). Migrating all tests will take a while, but we would already recommend trying to implement tests as Cucumber specifications. It should be relatively easy to convert these tests into the new syntax.

Hopefully this will motivate even more people from the broader Fineract community to participate in the project by sharing their domain specific knowledge as Cucumber specifications. Specifications are written in Englisch (although not a technical requirement).

When we refer to AsciiDoc then we mean the language or format that this documentation is written in. AsciiDoc is a markup language similar to Markdown (but more powerful and expressive) designed for technical documentation. You don’t need necessarily any specialized editors or tools to write your documentation in AsciiDoc, a plain text editor will do, but there are plenty of choices that give you a better experience (in this documentation we describe the basic usage with AsciiDoc plugins for IntelliJ, Eclipse and VSCode).

Asciidoctor on the other hand is the command line tool we use to transform documents written in AsciiDoc into HTML and PDF (Epub3 and Docbook are also available). There are three variants available:

Use backticks ` for text that should be monospaced, such as code or a class name in the body of a paragraph.

Longer code examples can be separated from text with source blocks.
These allow defining the syntax being used so the code is properly highlighted.

If your code block will include line breaks, put 4 hyphens (----) before and after the entire block.

AsciiDoc supports three types of lists:

Each type of list can be mixed with the other types. So, you could have an ordered list inside a labeled list if necessary.

There are two ways to include an image: inline or as a block. Inline images are those where text will flow around the image. Block images are those that appear on their own line, set off from any other text on the page. Both approaches use the image tag before the image filename, but the number of colons after image define if it is inline or a block. Inline images use one colon (image:), while block images use two colons (image::). Block images automatically include a caption label and a number (such as Figure 1). If a block image includes a title, it will be included as the text of the caption. Optional attributes allow you to set the alt text, the size of the image, if it should be a link, float and alignment. We have defined a global attribute {imagesdir} to standardize the location for all images (fineract-doc/src/docs/en/images).

Tables can be complex, but it is pretty easy to make a basic table that fits most needs.

AsciiDoc supports several types of callout boxes, called "admonitions":

It is enough to start a paragraph with one of these words followed by a colon (such as NOTE:). When it is converted to HTML, those sections will be formatted properly - indented from the main text and showing an icon inline.

You can add titles to admonitions by making it an admonition block. The structure of an admonition block is like this:

In this example, the type of admonition is included in square brackets ([NOTE]), and the title is prefixed with a period. Four equal signs give the start and end points of the note text (which can include new lines, lists, code examples, etc.).

We have set up the Ref Guide to be able to support STEM notation whenever it’s needed.

The AsciiMath syntax is supported by default, but LaTeX syntax is also available.

To insert a mathematical formula inline with your text, you can simply write:

MathJax.js will render the formula as proper mathematical notation when a user loads the page. When the above example is converted to HTML, it will look like this to a user: \$a//b\$

To insert LaTeX, preface the formula with latexmath instead of stem:

Long formulas, or formulas which should to be set off from the main text, can use the block syntax prefaced by stem or latexmath:

or for LaTeX:

See: plugins.jetbrains.com/plugin/7391-asciidoc

See: plugins.jetbrains.com/plugin/7017-plantuml-integration

See: marketplace.eclipse.org/content/asciidoctor-editor

See: plantuml.com/eclipse

See: marketplace.visualstudio.com/items?itemName=asciidoctor.asciidoctor-vscode

See: marketplace.visualstudio.com/items?itemName=jebbs.plantuml

A couple of secrets for third party services are automatically configured by the infrastructure team at The Apache Foundation for the Fineract Github account. At the moment this includes environment variables for:

See also:

It seems that Apache has some kind of org account or similar. Popped up a couple of times in the infrastructure documentation.

TBD

Other Fineract development related secrets, e. g. for deployments of demo systems on Google Cloud, AWS etc. are managed in a team account at 1Password. At the moment the following committers are members of the 1Password team account:

You can configure your GMail account and add another profile to use the Apache relay server if you need to send official messages. Please follow these instructions:

TBD.

See also: Send mail from another address without "on behalf of"

To be able to send emails via SMTP with your GMail account you probably need to create an app password. Please follow these instructions:

See also: Google Support: Sign in with App Passwords for more details.

TBD

The RM should, if one doesn’t already exist, first create a new release umbrella issue in JIRA. This issue is dedicated to tracking (a summary of) any discussion related to the planned new release. An example of such an issue is FINERACT-873 - Release Apache Fineract v1.4.0 RESOLVED.

The RM then creates an list of resolved issues & features through an initial check in JIRA for already resolved issues for the release, and then setup a timeline for release branch point. The time for the day the issue list is created to the release branch point must be at least two weeks in order to give the community a chance to prioritize and commit any last minute features and issues they would like to see in the upcoming release.

The RM must then send the pointer to the umbrella issue along with the tentative timeline for branch point to the developer lists. Any work identified as release related that needs to be completed should be added as a sub tasks of the umbrella issue to allow all developers and users to see the overall release progress in one place. The umbrella issue shall also link to any issues still requiring clarification whether or not they will make it into the release.

The RM should then inform users when the git branch is planned to be created, by sending an email based on this template:

Before a release is done, make sure that any issues that are fixed have their fix version setup correctly.

Move all unresolved JIRA issues which have this release as Fix Version to the next release

You can also run the following query to make sure that the issues fixed for the to-be-released version look accurate:

Finally, check out the output of the JIRA release note tool to see which tickets are included in the release, in order to do a sanity check.

Communicate with the community. You do not need to start a new email thread on the developer mailing list to notify that you are about to branch, just do it ca. 2 weeks after the initial email, or later, based on the discussion on the initial email.

You do not need to ask committers to hold off any commits until you have branched finished, as it’s always possible to fast-forward the branch to latest develop, or cherry-pick last minute changes to it. People should be able to continue working on the develop branch on bug fixes and great new features for the next release while the release process for the current release is being worked through.

You first need to close the release in JIRA so that the about to be released version cannot be used as "fixVersion" for new bugs anymore. Go to JIRA "Administer project" page and follow "Versions" in left menu. Table with list of all releases should appear, click on additional menu on the right of your release and choose "Release" option. Submit release date and you’re done.

Next, you create a git tag from the HEAD of the release’s git branch.

Create source and binary artifacts. Make sure to do some sanity checks. The tar and the release branch should match.

Make sure code compiles and tests pass on the uncompressed source.

All release artifacts must be signed. In order to sign a release you will need a PGP key. You should get your key signed by a few other people. You will also need to receive their keys from a public key server. See the Apache release signing page for more details. Please follow the steps defined in Release Sign.

Finally create a directory with release name (0.0.0-2b9de03b in this example) in dist.apache.org/repos/dist/dev/fineract and add the following files in this new directory:

Upload binary and source archives to ASF’s distribution dev (staging) area:

Following are the typical things we need to verify before voting on a release candidate. And the release manager should verify them too before calling out a vote.

Make sure release artifacts are hosted at dist.apache.org/repos/dist/dev/fineract

Voting has to be done on dev@fineract.apache.org. You can close the vote after voting period expires (72 hours) and you accumulate sufficient votes (minimum 3 x +1 PMC votes).

Upon receiving 3 x +1 from the PMC, or after 72 hours (whichever one comes first), reply to the voting thread and add the prefix "[RESULT]" to the subject line with the results, as follows:

In order to release you have to checkout release repository located on dist.apache.org/repos/dist/release/fineract and add release artifacts there.

You will now get an automated email from the Apache Reporter Service (no-reply@reporter.apache.org), subject "Please add your release data for 'fineract'" to add the release data (version and date) to the database on reporter.apache.org/addrelease.html?fineract (requires PMC membership).

As discussed in FINERACT-1154, now that everything is final, please do the following to remove the release branch (and just keep the tag), and make sure that everything on the release tag is merged to develop and that e.g. git describe works:

Finally update the fineract.apache.org website with the latest release details. The website’s HTML source code is available at github.com/apache/fineract-site.

Send an email to announce@apache.org (sender address must be @apache.org):