Chapter 2: Extract data from Jenkins CI

The Jenkins XML API is very useful for automating tasks like this – if you simply append

“/api/xml” to a Jenkins job URL, it will serve up an XML version (note there is also a JSON

API and a CLI and plenty of other options).

We will use curl application in order to run the api\xml and extract the data into xml files.

 

Step 1

Extracting JOBS DIMENSION DATA. This data will be the source of the JJOBS_DIM tables.

Option A:

Run curl and export the job dimension data into xml file.
curl -u ‘username:password’ ‘http://yourjenkinsserverip:8080/job/1.c%20Revert%20Cognos%20Machines%20(Main%20CI)/api/xml’ -o ‘C:\Adi Docs\Jenkins Data\1cRevertCognosMachinesMainCI.xml

Option B:

Run sql insert command

insert into JJOBS_DIM values (‘1.c Revert Cognos Machines (Main CI)’,’1.c Revert Cognos Machines (Main CI)’,’multiJobProject’,’http://yourjenkinsserverip:8080/job/1.c%20Revert%20Cognos%20Machines%20(Main%20CI)/’);

 

Step 2

Extracting JOBS DAILY FACT DATA. This data will be the source of the daily jobs fact tables.

 

curl -u ‘username:password’ ‘http://1.9.20.105:8080/job/1.c%20Revert%20Cognos%20Machines%20(Main%20CI)/lastBuild/api/xml’ -o ‘C:\Adi Docs\Jenkins Data\1cRevertCognosMachinesMainCIlastbuild.xml

Step 3

Extracting JOBS DAILY FACT DATA. This data will be the source of the daily jobs fact tables.

 

curl -u ‘username:password’ –globoff ‘http://yourjenkinsserverip:8080/job/1.c%20Revert%20Cognos%20Machines%20(Main%20CI)/lastBuild/api/xml?depth=1&tree=displayName,fullDisplayName,id,actions%5Bcauses%5BupstreamProject,upstreamBuild,shortDescription%5D%5D’ -o ‘C:\Adi Docs\Jenkins Data\1cRevertCognosMachinesMainCIrellastbuild.xml

Step 4

Extracting PIPELINE DIM DATA. This data will be the source of the pipeline dimension table. This table will store pipeline data and jobs related to it.

curl -u ‘username:password’ ‘http://yourjenkinsserverip:8080/job/1.c%20Revert%20Cognos%20Machines%20(Main%20CI)/api/xml’ -o ‘C:\Adi Docs\Jenkins Data\1cRevertCognosMachinesMainCI.xml’

 

 

Please pay attention to the files:

  1. Regular XXX.xml files are the source of the dimension table JJBOS_DIM
  2. lastbuild.xml files are the source of all the detailed data regarding our last day job run. These files will be loaded into JLASTJOBS_FACT table
  3. rellastbuild.xml files are the source of all the data regarding the connections (upstream, downstream)  between specific jobs run in a specific daily pipeline. These files will be loaded into JPIPELINEJOBS table.
  4. pipeline files are the source of the JPIPELINE_DIM table.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s