PyDFix

About

PyDFix is a tool that helps detect and fix dependency errors that cause the unreproducibility of Python builds. PyDFix takes as input the current build log, the original build log and the source code. PyDFix first identifies dependency errors and possible dependency packages causing these errors using LogErrorAnalyzer. This is followed by iteratively building a patch that makes the build reproducible again by Itera- tiveDependencySolver. The iterative algorithm for building the patch keeps re-running the build with intermediate patches and analyzing the new build logs produced to further identify errors and problematic dependency version specifications. This process continues until the build becomes reproducible, or all patch options have been tested and deemed not useful.

Getting Started

We recommend the use of a terminal multiplexer like tmux to run the processes for PyDFix. This is to prevent the processes from being interrupted based on connection to the terminal. The tmux command is included in Step 2 of setup.

Setup using Docker image (recommended)

🔶Requirement: Docker version >= 20.10.3 Ubuntu version = 18.04 🔶 The estimated running time for the following commands is: ≤15 minutes.

Step 1: Pull PyDFix docker image

$ sudo docker pull ucdavisplse/pydfix

Step 2: Run container from PyDFix image

$ tmux
$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --net=host -it ucdavisplse/pydfix

Step 3: Setup Docker permissions

$ source docker_setup.sh
$ sudo su pydfix

Step 4: Setup Datasets

$ export PATH=$PATH:/home/pydfix/PyDFix/BugsInPy/framework/bin/
$ source bugsinpy_setup.sh
$ source bugswarm_setup.sh

Step 5: Setup GitHub credentials

This step will require your GitHub username and personal access token.

$ source setup_credentials.sh

PyDFix can also be set up without using the Docker image, the instructions are detailed in this section.

Detailed Description

The replication process is broadly divided into 4 sections as follows:

• Step 1: BugSwarm Metrics First we gather data about the frequency of dependency package usage in BugSwarm builds as a part of our motivation. In this step, we will re-create the data presented in Section 3.1, Figures 2 and 3 of the paper.
• Step 2: LogErrorAnalyzer Next we use the first component of PyDFix, LogErrorAnalyzer which analyzes current build logs and on comparing them with original build logs identifies builds that are unreproducible due to dependency errors. Here, the script will run LogErrorAnalyzer which is described in further detail in Section 4.1 and generate the data produced in Table 4 of the paper.
• Step 3: IterativeDependencySolver The next component of PyDFix is IterativeDependencySolver which takes as input the builds identified in the previous step and iteratively computes a patch comprising of a list of pinned dependencies to make the build reproducible again. This step runs IterativeDependencySolver which is described in detail in Section 4.2 on all the artifacts identified in Step 2 to generate the data presented in Table 5 of the paper.
• Step 4: BugsSwarm Patch Metrics Finally, we gather metrics on the patches computed by PyDFix. In this step, the script will re-create the data shown in Table 6 of the paper. <br/>  

🔴 Note: While running any of the commands associated with any step, errors messages maybe printed to the terminal which are encountered while using a build from the datasets used. Such error messsages DO NOT indicate that the command was unsuccessful. Each step prints a message at the end of the run showing the details of the run, and summarizing its results. This message will indicate that the step was successful.

In the section Reusing PyDFix we have provided a high level overview of the modifications required to extend the work done so far.

STEP 1: BugSwarm Metrics

The following script gathers metrics about dependency packages from original build logs of BugSwarm artifacts. (Recall that we could not perform this on BugsInPy due to the absence of original build logs.)<br/> 🔶 The estimated running time for the following command is: ≤10 minutes.

$ python3 gather_metrics_from_orig_logs.py -path ~/PyDFix -originallogs ~/PyDFix/orig_log_bugswarm

Note: The artifacts.json for BugSwarm v1.1.3 is included in the root of the cloned directory.

Output:

This generates a CSV file named orig_log_metrics.csv which has the following columns.

| Sl | Column Name | Column Description | |----|-------------------------------------------|-----------------------------------------------------------------------------| | 1 | Artifact Image Tag | Image tag of artifact | | 2 | Pass/Fail | Passed or Failed build artifact | | 3 | Project Dependency Count | Total number of dependencies declared within the source code | | 4 | Pinned Dependency Count | Total number of pinned dependencies declared within the source code | | 5 | Constrained Dependency Count | Total number of constrained dependencies declared within the source code | | 6 | Unconstrained Dependency Count | Total number of unconstrained dependencies declared within the source code | | 7 | Transitive Dependency Count | Total number of transitive dependencies required by the build | | 8 | Transitive Pinned Dependency Count | Total number of pinned transitive dependencies required by the build | | 9 | Transitive Constrained Dependency Count | Total number of constrained transitive dependencies required by the build | | 10 | Transitive Unconstrained Dependency Count | Total number of unconstrained transitive dependencies required by the build | | 11 | Installed Packages | Total packages required by the build |

STEP 2: LogErrorAnalyzer

This component of PyDFix runs current build of artifacts from BugSwarm and BugsInPy and identifies whether the current build has dependency errors that did not exist in the original build. Since BugsInPy does not contain original build logs, we use the logs generated by running BugsInPy commands. The detailed technical approach for LogErrorAnalyzer is described in Section 4.1 and this section reproduces the data described in Section 5.2 and Table 4 of the paper. <br/>  

🔴 Note: For each dataset, we have prepared a representative subset of each dataset to perform a shorter run. Here we mean that the builds included in the subset have final fix outcomes which is in representative proportion of PyDFix's fix outcomes in the original experiments described in the paper.  

🔴 Note: The subsets are created purely for the convenience of the evaluator. If the evaluator wishes they can perform the complete Full Runs the commmands for which are also provided in segments following the Subset Runs.  

🔴 Note: If the evaluator chooses the perform Subset Runs in this step, the following steps will automatically work on subsets. Step 3 which runs IterativeDependencySolver only works on builds identified by LogErrorAnalyzer and the patch metrics collected in Step 4 will consider the patches generated in Step 3. Thus, choosing to do a subset run in this step will automatically make the future steps consider builds identified from the subset.

BugSwarm

Subset Run

This runs LogErrorAnalyzer on a subset of 20 BugSwarm builds, which is a representative sample of the entire dataset.<br/> 🔶 The estimated running time for the following command is: ≤15 minutes.

$ mkdir repro_log_bugswarm
$ python3 bugswarm_log_dependency_analyzer.py -path ~/PyDFix/repro_log_bugswarm -originallogs ~/PyDFix/orig_log_bugswarm -subsetrun

Full Run

This runs LogErrorAnalyzer on all available BugSwarm builds which have original source code available on GitHub.<br/> 🔶 The estimated running time for the following command is: ≤3 hours.

$ mkdir repro_log_bugswarm
$ python3 bugswarm_log_dependency_analyzer.py -path ~/PyDFix/repro_log_bugswarm -originallogs ~/PyDFix/orig_log_bugswarm

The output from running these commands is a message printed to the terminal and a file artifacts_dependency_broken.csv. These are explained in the LogErrorAnalyzer Output Section. <br/>

Note: If the LogErrorAnalyzer run does not complete successfully, there may be Docker containers left running which will interfere with subsequent runs. To resolve this issue and cleanup running containers, run the following command.

$ source bugswarm_cleanup_containers.sh

BugsInPy

Subset Run

This runs LogErrorAnalyzer on a subset of 20 BugsInPy builds, which is a representative sample of the entire dataset.<br/> 🔶 The estimated running time for the following command is: ≤30 minutes.

$ mkdir repro_log_bugsinpy 
$ python3 bugsinpy_log_dependency_analyzer.py -path ~/PyDFix/repro_log_bugsinpy -originallogs ~/PyDFix/orig_log_bugsinpy -component ~/PyDFix/BugsInPy -subsetrun

Full Run

This runs LogErrorAnalyzer on