Download docs/source/cicd.rst from ineso22/Mozdef: direct link, hf CLI and curl.
- Browser
- Download file 10.7 kB
-
https://huggingface.co/ineso22/Mozdef/resolve/main/docs/source/cicd.rst
- Command line
-
hf download hf://ineso22/Mozdef/docs/source/cicd.rst
-
curl -L -o cicd.rst https://huggingface.co/ineso22/Mozdef/resolve/main/docs/source/cicd.rst
10.7 kB
| Continuous Integration and Continuous Deployment | |
| ================================================ | |
| Overview | |
| -------- | |
| Each git commit to the `master` branch in GitHub triggers both the TravisCI | |
| automated tests as well as the AWS CodeBuild building. Each git tag applied to a | |
| git commit triggers a CodeBuild build. | |
| Travis CI | |
| --------- | |
| Travis CI runs tests on the MozDef code base with each commit to `master`. The | |
| results can be seen on the | |
| `Travis CI MozDef dashboard <https://travis-ci.org/mozilla/MozDef/>`_ | |
| The Test Sequence | |
| _________________ | |
| * Travis CI creates webhooks when first setup which allow commits to the MozDef | |
| GitHub repo to trigger Travis. | |
| * When a commit is made to MozDef, Travis CI follows the instructions in the | |
| `.travis.yml <https://github.com/mozilla/MozDef/blob/master/.travis.yml>`_ | |
| file. | |
| * `.travis.yml` installs `docker-compose` in the `before_install` phase. | |
| * In the `install` phase, Travis runs the | |
| `build-tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L88-L89>`_ | |
| make target which calls `docker-compose build` on the | |
| `docker/compose/docker-compose-tests.yml`_ file which builds a few docker | |
| containers to use for testing. | |
| * In the `script` phase, Travis runs the | |
| `tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L52>`_ | |
| make target which | |
| * calls the `build-tests` make target which again runs `docker-compose build` | |
| on the `docker/compose/docker-compose-tests.yml`_ file. | |
| * calls the | |
| `run-tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L67-L69>`_ | |
| make target which. | |
| * calls the | |
| `run-tests-resources <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L60-L61>`_ | |
| make target which starts the docker | |
| containers listed in `docker/compose/docker-compose-tests.yml`_. | |
| * runs `flake8` with the | |
| `.flake8 <https://github.com/mozilla/MozDef/blob/master/.flake8>`_ | |
| config file to check code style. | |
| * runs `py.test tests` which runs all the test cases. | |
| AWS CodeBuild | |
| ------------- | |
| Enabling GitHub AWS CodeBuild Integration | |
| _________________________________________ | |
| Onetime Manual Step | |
| ******************* | |
| The steps to establish a GitHub CodeBuild integration unfortunately | |
| require a onetime manual step be done before using CloudFormation to | |
| configure the integration. This onetime manual step **need only happen a | |
| single time for a given AWS Account + Region**. It need **not be | |
| performed with each new CodeBuild project or each new GitHub repo** | |
| 1. Manually enable the GitHub integration in AWS CodeBuild using the | |
| dedicated, AWS account specific, GitHub service user. | |
| 1. A service user is needed as AWS CodeBuild can only integrate with | |
| GitHub from one AWS account in one region with a single GitHub | |
| user. Technically you could use different users for each region in | |
| a single AWS account, but for simplicity limit yourself to only | |
| one GitHub user per AWS account (instead of one GitHub user per | |
| AWS account per region) | |
| 2. To do the one time step of integrating the entire AWS account in | |
| that region with the GitHub service user | |
| 1. Browse to `CodeBuild`_\ in AWS and click Create Project | |
| 2. Navigate down to ``Source`` and set ``Source Provider`` to | |
| ``GitHub`` | |
| 3. For ``Repository`` select | |
| ``Connect with a GitHub personal access token`` | |
| 4. Enter the persona access token for the GitHub service user. If | |
| you haven't created one do so and grant it ``repo`` and | |
| ``admin:repo_hook`` | |
| 5. Click ``Save Token`` | |
| 6. Abort the project setup process by clicking the | |
| ``Build Projects`` breadcrumb at the top. This “Save Token” | |
| step was the only thing you needed to do in that process | |
| Grant the GitHub service user access to the GitHub repository | |
| ************************************************************* | |
| 1. As an admin of the GitHub repository go to that repositories | |
| settings, select Collaborators and Teams, and add the GitHub | |
| service user to the repository | |
| 2. Set their access level to ``Admin`` | |
| 3. Copy the invite link, login as the service user and accept the | |
| invitation | |
| Deploy CloudFormation stack creating CodeBuild project | |
| ****************************************************** | |
| Deploy the ``mozdef-cicd-codebuild.yml`` CloudFormation template | |
| to create the CodeBuild project and IAM Role | |
| .. _CodeBuild: https://us-west-2.console.aws.amazon.com/codesuite/codebuild/ | |
| The Build Sequence | |
| __________________ | |
| * A branch is merged into `master` in the GitHub repo or a version git tag is | |
| applied to a commit. | |
| * GitHub emits a webhook event to AWS CodeBuild indicating this. | |
| * AWS CodeBuild considers the Filter Groups configured to decide if the tag | |
| or branch warrants triggering a build. These Filter Groups are defined in | |
| the ``mozdef-cicd-codebuild.yml`` CloudFormation template. Assuming the tag | |
| or branch are acceptable, CodeBuild continues. | |
| * AWS CodeBuild reads the | |
| `buildspec.yml <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/buildspec.yml>`_ | |
| file to know what to do. | |
| * The `install` phase of the `buildspec.yml` fetches | |
| `packer <https://www.packer.io/>`_ and unzips it. | |
| * `packer` is a tool that spawns an ec2 instance, provisions it, and renders | |
| an AWS Machine Image (AMI) from it. | |
| * The `build` phase of the `buildspec.yml` runs the | |
| `cloudy_mozdef/ci/deploy <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/deploy>`_ | |
| script in the AWS CodeBuild Ubuntu 14.04 environment. | |
| * The `deploy` script calls the | |
| `build-from-cwd <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L78-L79>`_ | |
| target of the `Makefile` which calls `docker-compose build` on the | |
| `docker-compose.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose.yml>`_ | |
| file, building the docker images in the AWS CodeBuild environment. These are | |
| built both so they can be consumed later in the build by packer and also | |
| for use by developers and the community. | |
| * `deploy` then calls the | |
| `docker-push-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L113>`_ | |
| make target which calls | |
| * the tag-images_ | |
| make target which calls the | |
| `cloudy_mozdef/ci/docker_tag_or_push tag <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_ | |
| script which applies a docker image tag to the local image that was just | |
| built by AWS CodeBuild. | |
| * the | |
| `hub-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L116-L117>`_ | |
| make target which calls the | |
| `cloudy_mozdef/ci/docker_tag_or_push push <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_ | |
| script which | |
| * Uploads the local image that was just built by AWS CodeBuild to DockerHub. | |
| If the branch being built is `master` then the image is uploaded both with | |
| a tag of `master` as well as with a tag of `latest`. | |
| * If the branch being built is from a version tag (e.g. `v1.2.3`) then the | |
| image is uploaded with only that version tag applied. | |
| * The `deploy` script next calls the | |
| `packer-build-github <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/Makefile#L34-L36>`_ | |
| make target in the | |
| `cloudy_mozdef/Makefile <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/Makefile>`_ | |
| which calls the | |
| `ci/pack_and_copy <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/pack_and_copy>`_ | |
| script which does the following steps. | |
| * Calls packer which launches an ec2 instance, executing a bunch of steps and | |
| and producing an AMI | |
| * Shares the resulting AMI with the AWS Marketplace account | |
| * Copies the resulting AMI to a list of additional AWS regions | |
| * Copies the tags from the original AMI to these copied AMIs in other regions | |
| * Shares the AMIs in these other regions with the AWS Marketplace account | |
| * Creates a blob of YAML which contains the AMI IDs. This blob will be used | |
| in the CloudFormation templates | |
| * When `ci/pack_and_copy` calls packer, packer launches an ec2 instance based on | |
| the configuration in | |
| `cloudy_mozdef/packer/packer.json <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/packer/packer.json>`_ | |
| * Within this ec2 instance, packer `clones the MozDef GitHub repo and checks | |
| out the branch that triggered this build | |
| <https://github.com/mozilla/MozDef/blob/c7a166f2e29dde8e5d71853a279fb0c47a48e1b2/cloudy_mozdef/packer/packer.json#L58-L60>`_. | |
| * Packer replaces all instances of the word `latest` in the | |
| `docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_ | |
| file with either the branch `master` or the version tag (e.g. `v1.2.3`). | |
| * Packer runs `docker-compose pull` on the | |
| `docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_ | |
| file to pull down both the docker images that were just built by AWS | |
| CodeBuild and uploaded to Dockerhub as well as other non MozDef docker | |
| images. | |
| * After packer completes executing the steps laid out in `packer.json` inside | |
| the ec2 instance, it generates an AMI from that instance and continues with | |
| the copying, tagging and sharing steps described above. | |
| * Now back in the AWS CodeBuild environment, the `deploy` script continues by | |
| calling the | |
| `publish-versioned-templates <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/Makefile#L85-L87>`_ | |
| make target which runs the | |
| `ci/publish_versioned_templates <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/publish_versioned_templates>`_ | |
| script which | |
| * injects the AMI map yaml blob produced | |
| earlier into the | |
| `mozdef-parent.yml <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/cloudformation/mozdef-parent.yml#L86-L87>`_ | |
| CloudFormation template so that the template knows the AMI IDs of that | |
| specific branch of code. | |
| * uploads the CloudFormation templates to S3 in a directory either called | |
| `master` or the tag version that was built (e.g. `v1.2.3`). | |
| .. _docker/compose/docker-compose-tests.yml: https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-tests.yml | |
| .. _tag-images: https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L109-L110 | |