Licensing Check Scripts

This directory holds various scripts to aid in clearing IP on files. The main script is log2json which should receive a path to a file (relative to repository root). It will retrieve the file history and build a JSON output with all metadata needed for analysis. The second script is check.py which receives a JSON file generated from the previous tool (either from a file or from stdin, using - for the filename).

The check script will:

1. retrieve git commit authors
2. parse commit message for possible attributions ("authored by: ...", among other variations)
3. retrieve file contents at each commit, parse the license header and try to extract authors and companies (copyrights) listed there

Steps 2 and 3 are based on heuristics. The attributions may not match the regular expressions used so there may be misdetections. Authors on headers are easier to detect. In fact, this will pick up various false positives (non-author strings) which will have to be ignored by the user.

All of these authorship information is aggregated and in a final step, the names are used to check for ICLAs, based on the ICLA databases (see below), which need to be manually downloaded. If a given author name is not matched, their email searched for in the author_mappings.json file, which is a dictionary of email to real name. This allows to handle users with alternative email addresses.

The script output will report a green check if author matched the ICLA database or a red cross if not. Note that given the false positives in steps 2 and 3, there may be both non-author strings that obviously do not match and also there may be an attribution which was not detected in a commit message. The thorough approach would be to run the check script with verbosity ('-v') which will print the metadata of each commit, including the commit message. If double verbosity is used ('-vv'), the whole file will be printed, which allows to check the header.

Inaccessible blobs

Since some files in the repositories lived during some part of their history in a separate repository (linked as a submodule to main repo), their blobs (basically the file at a given point in time) will not be accessible. This means that the file at that point in time cannot be accessed for analyzing its header.

Zero blob hash

Some blob hashes will be all zeros, which means that the file was deleted at this point in time. Sometimes this is due to merges or renames (which may be part of the moving in and out of submodules).

ICLA database

In order to retrieve a list of all users with CLAs, download the following files:

• https://whimsy.apache.org/public/icla-info.json
• https://whimsy.apache.org/public/icla-info_noid.json

There are two files since not all users with CLAs have Apache IDs. These lists do not contain emails, but a manual search form is also here:

• https://whimsy.apache.org/roster/committer/