About the DPDK Community Lab

Objective and Scope of the Project

The DPDK Community Lab is an open, independent testing resource for the DPDK project. Its purpose is to perform automated testing which ensures that the performance and quality of DPDK is maintained.

Hardware for the lab is provided by equipment vendors and is used to run a series of performance tests on new patch sets. The results of these tests are published on the results dashboard which is publicly accessible, and are also integrated into the DPDK instance of Patchwork. The Community Lab does not publish raw performance numbers for vendor hardware, it only publishes deltas (% change) to flag bad DPDK patches. The results will be used by the project's maintainers to help determine whether new patch sets should be accepted into the project.

The initial goal of the DPDK Community Lab is:

1. Identify any regression in DPDK performance. The DPDK Community Lab will host equipment provided by multiple Members to UNH-IOL and run basic performance regression tests on new patches/patch sets on an automated basis. The aim of these tests is to determine if there has been any unexpected drop in DPDK performance as a result of recent changes.

In future, the following additional items may be added to the scope of the lab:

1. Identify any regression in the performance of DPDK-enabled applications. DPDK Members will be able to submit software applications to be run in the DPDK Community Lab. The details of how this can be managed may be complex, so this capability may be added at a later stage.
2. Demonstrate any new feature performance of DPDK. In each release, there may be some new performance optimizations or some new solutions. DPDK Members will be able to utilize the platforms in the performance test lab to show the new performance gains through DPDK-enabled applications.
3. The DPDK Community Lab may also be used as a training or demo lab for DPDK events.

DPDK Performance Test Lab Usage Guidelines

• Data generated by the DPDK Performance Test Lab shall not be used for competitive marketing purposes between DPDK Member equipment providers.
• Data from the test lab may be used to demonstrate and market the performance improvement of new DPDK features in a DPDK Member agnostic manner.

About the Dashboard

There are two main components to the Dashboard; the Results Database and the Dashboard itself. All results are uploaded via the Results Database REST API. The Dashboard also uses the REST API to populate its pages. The REST API can only be accessed by participating members. Meta information, such as the series title and patch submitter, utilize the Patchwork REST API.

Possible Patch Statuses

Apply Error

The patch series could not be applied

Build Error

The patch series could not be built

Downstream Failure

A downstream project could not be built.

Downstream Indeterminate

A downstream project's result could not be determined and could mean that the test did not run to completion

Indeterminate

A result could not be determined and could mean that the test did not run to completion

Not Applicable

The patch series is not active in Patchwork or is not required to be tested

Pass

All test results were within the tolerance threshold from the expected result

Pending

A tarball has not yet been generated for this patch series

Possible Regression

At least one test result was below the tolerance threshold from the expected result

Waiting

A tarball has been generated but no test results are available yet

Test Cases

Patch Details

The Dashboard calls groups of related patches a Patch Set. This is equivalent to Patchwork's Series. In the Results Database, the model mostly contains build information and result summaries about the series. It also contains a link to the series itself to allow for grabbing meta information about the series.

Times for patches and tests are displayed in UTC time.

Relative performance is measured by comparing the actual result with the baseline result. The baseline itself is saved on the test machine; only the delta results are submitted. The database has the capability of saving the baseline in order to calculate an absolute result, but that must be explicitly requested by the participating member.

Fluctuation in these results are expected, hence a tolerance is set to determine if the result passes or fails.

DPDK Community Lab Policy Guidelines

Latest Policy Document

Please note that some of the text has been taken from the DPDK Community Policies and Procedures document. If you would like to make modifications to the text, please provide suggestions via the document.

Getting Involved

If an existing DPDK member is interested in providing hardware or software, contact the DPDK Community Lab CI at dpdklab@iol.unh.edu.

Priority support and installation is provided based on membership status in the DPDK project. If you're interested in becoming a DPDK member, visit the DPDK website to join.

Send bug reports or feature requests to the DPDK issue tracker.

For general CI questions and comments, contact the DPDK CI mailing list at ci@dpdk.org.

Requesting a Patch Retest

If CI reports a failure for a test label on Patchwork, it is possible to request a retest of a patch or series on those label(s). For a list of Patchwork testing labels, send an email reply on the patch or cover letter like so:

  Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-unit-arm64-testing, github-robot

Note that you may omit any labels which are not failing.

This is currently supported by the UNH-IOL Community Lab, and the GitHub Actions robot, so just those Patchwork testing labels beginning wih “iol-” or “github-robot”.