Tool Analysis
Brief overview
This section analyses some SBOM generation tools available in the market. The study is based on the fossa framework for Evaluating SBOM Tools, on the NTIA minimum element guidelines, and in some quality metrics for SBOMs available in the sbomqs tool. We also analyzed the SBOMs generated through knowledge graphs.
Tools overall analysis
Some of the features we used to analyze the SBOM generation tools were based on the framework for Evaluating SBOM Tools in fossa, such as:
- What standards it supports (SPDX, Cyclone DX)
- Compliance with the Cyber Security Executive Order
- Data Field Coverage
- Automation Support
- Programming Language Support
- Dependency Recognition Depth
We classified the tools following the SBOM Tool Classification Taxonomy document by the NTIA SBOM Formats & Tooling Working Group.
The SBOM generation tools we analyzed are listed below, and their overall analysis can be found here.
- Syft
- Trivy
- Bom
- Microsoft/sbom-tool
- Sbom4python
- gh-sbom
- CycloneDX Generator
- Tern
- ScanCode toolkit
- OSS Review Toolkit
- Cosign
- Augur
- SPDX SBOM Generator
- SwiftBOM
Quality metrics to assess the generated SBOMs
We evaluated the performance of the tools by examining the quality of the SBOMs they produced for some case studies projects:
- Case Study 1: The target is the PyTorch repo
- Case Study 2: The target is the TheBloke/text-generation-webui repo
The SBOMs generated by the tools were analyzed using the sbomqs tool. The analysis results can be found on the pages dedicated to each case study.
The evaluation of SBOMs involves five categories in the scoring system, NTIA-minimum-elements, Structural, Semantic, Quality, and Sharing. The score is calculated based on the number of elements in each category, and the higher the score, the more consumable the SBOMs should be. More details about the sbomqs SBOM quality check can be found here.
The following table shows the features used to calculate each category’s score.
NTIA-minimum-elements
Includes features that help you understand if your SBOM complies with NTIA minimum element guidelines.
| Feature | Description |
|---|---|
| comp_with_name | number of components that have names divided by the total components detected |
| comp_with_supplier | number of components that have supplier divided by the total components detected |
| comp_with_uniq_ids | number of components that have unique ID’s divided by the total components detected |
| comp_with_version | number of components that have versions divided by the total components detected |
| sbom_authors | doc has authors |
| sbom_creation_timestamp | doc has creation timestamp |
| sbom_dependencies | doc has relationships |
Structural
It checks if the SBOM complies with the underlying specifications, be it SPDX or CycloneDX
| Feature | Description |
|---|---|
| sbom_parsable | provided sbom is parsable |
| sbom_spec | provided sbom is in a supported sbom format of spdx, cyclonedx |
| sbom_spec_file_format | provided sbom should be in supported file format for spec: json and version: json, yaml, rdf, tag-value |
| sbom_spec_version | provided sbom should be in supported spec version for spec: SPDX-2.3 and versions: SPDX-2.1, SPDX-2.2, SPDX-2.3 |
Semantic
It checks the meaning of SBOM fields specific to their standard.
| Feature | Description |
|---|---|
| comp_with_checksums | number of components that have checksums divided by the total components detected |
| comp_with_licenses | number of components that have licenses divided by the total components detected |
| sbom_required_fields | Doc Fields:(true or false) Pkg Fields:(true or false) |
Quality
It helps determine the quality of the data present in the sbom.
| Feature | Description |
|---|---|
| comp_valid_licenses | number of components that have valid licenses divided by the total components detected |
| comp_with_any_vuln_lookup_id | number of components that have any lookup id divided by the total components detected |
| comp_with_deprecated_licenses | number of components that have deprecated licenses divided by the total components detected |
| comp_with_multi_vuln_lookup_id | number of components that have multiple lookup id divided by the total components detected |
| comp_with_primary_purpose | number of components that have primary purpose specified divided by the total components detected |
| comp_with_restrictive_licenses | number of components that have restricted licenses divided by the total components detected |
| sbom_with_creator_and_version | number of components that have creator and version divided by the total components detected |
Other tools used to support the analysis
- SBOM Validation tool: SPDX Online Tools
- SBOM Conversion tool: pyspdxtools