Performance

Information

These are the results of our performance tests. If your performance is sensitive and/or you are working on server sizing, it may be better that you perform performance tests on a staging instance yourself.


Summary

Basically, once you have 50.000 requirements in the database, expect 20ms per requirement on the page when you save a page, and 20ms per requirement on view. This is not a commitment, as it depends on the machine, the set up, the configuration, the latency of the DB, the version of Confluence and Requirement Yogi.

Bottlenecks

We recommend:

  • 150 requirements per page, with a maximum of 400 (Confluence doesn't support infinite pages anyway).

  • The default Global Limit is 12,000 and it should be the optimum for most people. If administrators notice that the Confluence is running slow, this might be where you need to check first.

  • The expected number of dependencies, as explained on Limitations, is about 20. We don't have a hard limit, but please don't design a model with 1000-to-1 relationships, whether on requirements or on Jira issues.

  • The size limit for baselines depends on the Global Limit and it is 12,000 requirements by default.

Processes to monitor:

  • For administrators, the process to monitor is baseline creation. If baselines take more than 30s to create, they load very important amounts of data in memory, such as requirements and the associated pages, and hold on to the transaction until the baseline is created.

    • Symptom: If baselines are too big, they will cause a memory error, which symptom is to slow the instance down until it doesn't process anything anymore (the phase where the JVM tries to garbage-collect forever) until, at one point, it throws a HeapSpaceException or OutOfMemoryError.

    • Resolution: Administrators can reduce the baseline size by reducing the Global Limit.

    • Our plans: In RY-965, we will ensure we communicate it to users in the UI, and in RY-966 we will make it a background task chopped in smaller transactions.

  • The disk space usage in Jira. Delete history in the administration if the AO_42D05A_RY_AUDIT_TRAIL table becomes too large. See Maintenance guide.

  • The queue. See Queue performance.

Performance in 3.0

Changes in 3.0:

  • When saving a page, we've gone from 3500 requests to 1356 (Don't assume we just had to switch a flag, it instead required heavy optimizations for each request).

  • We have reworked the data model around dependencies, it shouldn't change the performance very much.

We have evaluated on a personal machine with the following setup:

  • iMac 2013 – 3.4 GHz Intel Core i5,

  • RAM 8GB.

  • Database latency: 5 to 10ms (random) per query.

  • Network latency to the database: 2ms (simulated) per query,

  • Database prefilled with 80.000 requirements.

We have simply instrumented the code and created massive pages:

Event

Time (in addition to Confluence's algorithm).

For ~400 requirements, ~525Kb text per page, 2ms network latency. No Jira connection.

Time

1ms latency,
1000 requirements,
1.3 MB page size.

Event

Time (in addition to Confluence's algorithm).

For ~400 requirements, ~525Kb text per page, 2ms network latency. No Jira connection.

Time

1ms latency,
1000 requirements,
1.3 MB page size.

Page creation

  • 5483ms saving the page

  • 12ms per requirement,

  • Total: +5 seconds (on top of Confluence's algorithm).

  • 8.3s (Confluence)

  • 7ms per requirement

  • Total +7.5s

Submission of excerpts

(This operation is in the background, the user doesn't wait for this).

  • 14ms per modified requirement
    (depending on the size of text and the number of properties).

  • Total 5.65s for 400 modified requirements.

  • Note: This is with 2ms additional network latency per DB queries. For most networks between a server and its database, the latency is sub-1ms.

  • 7ms per modified requirement.

  • Total +7854ms

Tested for Requirement Yogi 3.0.0 / C7.4 with 2ms and 1ms network latency,
in addition to the database latency, already loaded with 80.000 requirements.

Performance in 2.6.9

Same conditions:

Event

Time (in addition to Confluence's algorithm).

For ~400 requirements, ~525Kb text per page, 2ms network latency. No Jira connection.

Time

1ms latency,
1000 requirements,
1.3 MB page size.

Event

Time (in addition to Confluence's algorithm).

For ~400 requirements, ~525Kb text per page, 2ms network latency. No Jira connection.

Time

1ms latency,
1000 requirements,
1.3 MB page size.

Page creation

  • 8338ms saving the page

  • 20ms per requirement,

  • Total: +7.9 seconds (on top of Confluence's algorithm).

  • 12.9s
    saving (Confluence)

  • 12ms per requirement

  • Total +12.1s

Submission of excerpts

(This operation is in the background, the user doesn't wait for this).

  • 28ms per modified requirement
    (depending on the size of text and the number of properties).

  • Total 11.40s for 400 modified requirements.

  • Note: This is with 2ms additional network latency per DB queries. For most networks between a server and its database, the latency is sub-1ms.

  • 14ms per modified requirement.

  • Total +13.9s

Tested for Requirement Yogi 2.6.9 / C7.4 with 2ms and 1ms network latency,
in addition to the database latency, already loaded with 80.000 requirements.

Performance in 2.0

Same conditions:

Event

Time (in addition to Confluence's algorithm).

For ~400 requirements, ~525Kb text per page, 2ms network latency. No Jira connection.

Time

1ms latency,
1000 requirements

Event

Time (in addition to Confluence's algorithm).

For ~400 requirements, ~525Kb text per page, 2ms network latency. No Jira connection.

Time

1ms latency,
1000 requirements

Page creation

  • 480ms rendering of the page

  • 22ms per requirement,

  • Total: +8.9 seconds (on top of Confluence's algorithm).

  • 1082ms
    rendering (Confluence)

  • 13ms per requirement

  • Total +13s

Page edition

  • 700ms rendering of the page

  • 21ms per requirement,

  • Total +11.6 seconds.

  • 1700ms rendering

  • 15ms per requirement

  • Total +15.7s

Submission of excerpts

(This operation is in the background, the user doesn't wait for this).

  • 20-40ms per modified requirement
    (depending on the size of text and the number of properties).

  • Total 18,7s for 400 modified requirements.

  • Note: This is with 2ms additional network latency per DB queries. For most networks between a server and its database, the latency is sub-1ms.

  • 8ms per modified requirement.

  • Total +8111ms

New result:

  • 3ms per modified requirement

  • Total: 3s

Tested for Requirement Yogi 2.0.0 with 2ms and 1ms network latency, in addition to the database latency, already loaded with 80.000 requirements.

Performance changes in 2.0:

  • We've deeply modified the indexing algorithm in 2.0, because we are now importing Excel files.

  • This algo generally reads first and checks whether it needs to change the data, instead of deleting all and writing blindly.

  • We did not notice much change in speed. Some speeds are improved, other are worse, depending on the number of modified requirements and properties.

Performance in 1.11.5

  • For pages with no requirements, we've improved the speed by skipping our indexation:

    • We skip the parsing if the storage format hasn't changed,

    • We skip the parsing if the rendered format hasn't changed, in case it contains an "Include" or "Scaffolding" macro.

    • We skip the parsing if there is no requirement in the old or new version.

  • For pages with requirements:

    • We've added indexes on database columns. On our instance we get 5x faster results when saving a page, but we may be in special circumstances.

    • When we index a page (=when a user saves a page), we've batched the lookups of requirements, so we don't do 1 database request for each requirement on the page. On our instance, we get again 4x faster times depending on database latency (most LANs are on 1ms latency, but we've measured with 5ms).

    • We'd be thrilled if you have 20x better response times than in 1.11.4, but we'll check back with customers before asserting that.

Errors in the logs?

If your server meets problems like "OutOfMemoryException", "Java Heap Space" or "SiteMesh" exceptions, it could be related to building the Traceability matrix. One important thing to note is that the Requirement Yogi add-on may not be mentioned in those exceptions. If you are in this situation:

  • Use queries that return fewer requirements, in the search, but more importantly in the dependency, traceability and coverage matrix.

  • The dependency matrix and the coverage report can only be built by users who can export the space (This permission will change in version 2.0, see Release Notes 2.0).

  • Therefore, inform a subset of your users about the server limits and the tips to prevent a memory overflow; and restrict the export permissions to this group only.

  • As a last resort, don't use the coverage and dependency matrixes. You can always explore Confluence's database schema and run the queries using an SQL tool. Tables related to Requirement Yogi start with "AO_32F7CE_AOREQUIREMENT".

Also, please notify us if you're meeting performance issues. We've obtained Data Center certification with several performance tests, but real-life environment can be different than prepared tests and we want to keep improving.



Performance of upgrade tasks

Upgrade tasks are executed once, when the addon is upgraded to the provided version.

Version

Name

Execution Time

Memory usage

Assumption

Goal of the upgrade

Version

Name

Execution Time

Memory usage

Assumption

Goal of the upgrade

1.x

Upgrade tasks performance was not measured before version 2.0.

2.0.1

V46RemoveOrphanRequirementsUpgradeTask

2 minutes 21 seconds

Max 4.1MB of local variables at any given time.

1,000,000 active requirements

1KB of text per requirement

iMac 2013 – 3.4 GHz Intel Core i5

Some requirements are in "ACTIVE" status despite having no page attached. Mark them as "DELETED".

2.2.2

V47EnsureEstateIsNotNullUpgradeTask

12 seconds

– Negligible

100,000 links to Jira

For the integration with external tools, we assume there is always a "space key", which is always the same value for Jira.

After 2.2.2, all upgrade tasks are non-blocking background jobs, executed 3 minutes after plugin installation.