Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parse botocore.endpoint logs into a list of AWS actions #49312

Merged
merged 15 commits into from
Mar 18, 2019

Conversation

s-hertel
Copy link
Contributor

@s-hertel s-hertel commented Nov 29, 2018

Add an option to parse botocore.endpoint logs for the AWS actions performed during a task

Add a callback to consolidate all AWS actions used by modules

Allow integration tests to output AWS actions

Added some docs to help contributors help with the needs_ci_update queue

SUMMARY

This helps remove some friction when determining the minimum permissions needed for integration tests, as well as being a step in the direction of being able to document #22552

Please bikeshed names.

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

@s-hertel s-hertel requested a review from mattclay November 29, 2018 16:18
@ansibot
Copy link
Contributor

ansibot commented Nov 29, 2018

Hi @s-hertel, thank you for submitting this pull-request!

click here for bot help

@ansibot
Copy link
Contributor

ansibot commented Nov 29, 2018

@ansibot ansibot added affects_2.8 This issue/PR affects Ansible v2.8 aws cloud core_review In order to be merged, this PR must follow the core review workflow. feature This issue/PR relates to a feature request. module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. new_plugin This PR includes a new plugin. support:community This issue/PR relates to code supported by the Ansible community. support:core This issue/PR relates to code supported by the Ansible Engineering Team. test This PR relates to tests. labels Nov 29, 2018
@ansibot
Copy link
Contributor

ansibot commented Nov 29, 2018

The test ansible-test sanity --test pylint [explain] failed with 1 error:

lib/ansible/module_utils/aws/core.py:151:0: anomalous-backslash-in-string Anomalous backslash in string: '\.'. String constant might be missing an r prefix.

The test ansible-test sanity --test pep8 [explain] failed with 1 error:

lib/ansible/module_utils/aws/core.py:151:29: W605 invalid escape sequence '\.'

click here for bot help

@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed core_review In order to be merged, this PR must follow the core review workflow. labels Nov 29, 2018
@samdoran samdoran removed the needs_triage Needs a first human triage before being processed. label Dec 4, 2018
@ansibot
Copy link
Contributor

ansibot commented Dec 4, 2018

The test ansible-test sanity --test pylint [explain] failed with 1 error:

lib/ansible/module_utils/aws/core.py:151:0: anomalous-backslash-in-string Anomalous backslash in string: '\.'. String constant might be missing an r prefix.

The test ansible-test sanity --test pep8 [explain] failed with 1 error:

lib/ansible/module_utils/aws/core.py:151:29: W605 invalid escape sequence '\.'

click here for bot help

@ansibot
Copy link
Contributor

ansibot commented Dec 12, 2018

@ansibot ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Dec 12, 2018
@mattclay mattclay added the ci_verified Changes made in this PR are causing tests to fail. label Dec 19, 2018
@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed ci_verified Changes made in this PR are causing tests to fail. stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Jan 12, 2019
test/runner/lib/executor.py Outdated Show resolved Hide resolved
test/integration/targets/route53/tasks/main.yml Outdated Show resolved Hide resolved
lib/ansible/plugins/callback/aws_resource_actions.py Outdated Show resolved Hide resolved
def exit_json(self, *args, **kwargs):
if self.params.get('debug_botocore_endpoint_logs'):
kwargs['resource_actions'] = self._get_resource_action_list()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactor these two lines into a function and call it from the exit_json, fail_json and fail_json_aws.

Maybe just move the logic into _get_resource_action_list and rename it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I refactored but didn't move the logic into _get_resource_action_listbecause it was intended to collect unique actions an individual task uses. Not strictly necessary for collecting actions a given playbook uses, but I thought having the finer grain of control had merit.

@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed core_review In order to be merged, this PR must follow the core review workflow. labels Jan 15, 2019
@s-hertel s-hertel force-pushed the collect_aws_actions branch from 46bfef9 to 0a7e277 Compare March 1, 2019 17:42
@@ -11,6 +11,7 @@
- include: ./credential_tests.yml
- module_defaults:
group/aws:
debug_botocore_endpoint_logs: True
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once #51149 is merged I'll be able to remove this. Right now any tests setting module_defaults for group/aws (such as this one) overwrite the setting from the test runner.

@ansibot ansibot removed the needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html label Mar 1, 2019
@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Mar 2, 2019
Ensure AWS tests inherit environment

Remove AWS CLI in aws_rds inventory tests and use the module
@ansibot
Copy link
Contributor

ansibot commented Mar 6, 2019

@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed core_review In order to be merged, this PR must follow the core review workflow. labels Mar 6, 2019
@s-hertel
Copy link
Contributor Author

s-hertel commented Mar 6, 2019

bot_status

@ansibot
Copy link
Contributor

ansibot commented Mar 6, 2019

Components

hacking/aws_config/build_iam_policy_framework.py
support: core
maintainers:

lib/ansible/module_utils/aws/core.py
support: community
maintainers:

lib/ansible/module_utils/ec2.py
support: core
maintainers:

lib/ansible/modules/cloud/amazon/GUIDELINES.md
support: core
maintainers:

lib/ansible/plugins/callback/aws_resource_actions.py
support: community
maintainers:

lib/ansible/plugins/doc_fragments/aws.py
support: community
maintainers:

test/integration/targets/aws_eks/playbooks/full_test.yml
support: community
maintainers:

test/integration/targets/aws_eks/playbooks/old_version.yml
support: community
maintainers:

test/integration/targets/cloudformation_stack_set/playbooks/full_test.yml
support: community
maintainers: ryansb

test/integration/targets/ec2_instance/playbooks/full_test.yml
support: community
maintainers: Shaps ryansb

test/integration/targets/ec2_instance/playbooks/version_fail.yml
support: community
maintainers: Shaps ryansb

test/integration/targets/ec2_launch_template/playbooks/full_test.yml
support: community
maintainers: ryansb

test/integration/targets/ec2_launch_template/playbooks/version_fail.yml
support: community
maintainers: ryansb

test/integration/targets/ecs_cluster/playbooks/full_test.yml
support: community
maintainers: Java1Guy willthames

test/integration/targets/ecs_cluster/playbooks/network_assign_public_ip_fail.yml
support: community
maintainers: Java1Guy willthames

test/integration/targets/ecs_cluster/playbooks/network_fail.yml
support: community
maintainers: Java1Guy willthames

test/integration/targets/ecs_cluster/playbooks/network_force_new_deployment.yml
support: community
maintainers: Java1Guy willthames

test/integration/targets/ecs_cluster/playbooks/network_force_new_deployment_fail.yml
support: community
maintainers: Java1Guy willthames

test/integration/targets/efs/playbooks/full_test.yml
support: community
maintainers: akazakov ryansydnor

test/integration/targets/efs/playbooks/version_fail.yml
support: community
maintainers: akazakov ryansydnor

test/integration/targets/elb_target/playbooks/full_test.yml
support: community
maintainers: wimnat

test/integration/targets/elb_target/playbooks/version_fail.yml
support: community
maintainers: wimnat

test/integration/targets/elb_target_facts/playbooks/full_test.yml
support: community
maintainers: yaakov-github

test/integration/targets/inventory_aws_ec2/playbooks/populate_cache.yml
support: community
maintainers:

test/integration/targets/inventory_aws_ec2/playbooks/test_populating_inventory.yml
support: community
maintainers:

test/integration/targets/inventory_aws_ec2/playbooks/test_populating_inventory_with_constructed.yml
support: community
maintainers:

test/integration/targets/inventory_aws_rds/playbooks/populate_cache.yml
support: community
maintainers:

test/integration/targets/inventory_aws_rds/playbooks/test_populating_inventory.yml
support: community
maintainers:

test/integration/targets/inventory_aws_rds/playbooks/test_populating_inventory_with_constructed.yml
support: community
maintainers:

test/integration/targets/inventory_aws_rds/playbooks/test_refresh_inventory.yml
support: community
maintainers:

test/integration/targets/inventory_aws_rds/templates/inventory.yml
support: community
maintainers:

test/integration/targets/inventory_aws_rds/templates/inventory_with_cache.yml
support: community
maintainers:

test/integration/targets/inventory_aws_rds/templates/inventory_with_constructed.yml
support: community
maintainers:

test/runner/lib/cloud/aws.py
support: core
maintainers:

Metadata

waiting_on: ansible
changes_requested_by: null
needs_info: False
needs_revision: False
needs_rebase: False
merge_commits: []
too many files or commits: False
mergeable_state: clean
shippable_status: success
maintainer_shipits (module maintainers): 0
community_shipits (namespace maintainers): 0
ansible_shipits (core team members): 1
shipit_actors (maintainers or core team members): s-hertel
shipit_actors_other: []
automerge: automerge shipit test failed

click here for bot help

@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Mar 6, 2019
minimize and clarify

revise testing procedures

moves troubleshooting into its own section

this is cleaner

bullets for the two choices

make requirements clearer
@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed core_review In order to be merged, this PR must follow the core review workflow. labels Mar 13, 2019
@s-hertel
Copy link
Contributor Author

!needs_revision

@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Mar 14, 2019
@s-hertel s-hertel merged commit 7da565b into ansible:devel Mar 18, 2019
@ansible ansible locked and limited conversation to collaborators Jul 25, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.8 This issue/PR affects Ansible v2.8 aws cloud core_review In order to be merged, this PR must follow the core review workflow. feature This issue/PR relates to a feature request. module This issue/PR relates to a module. new_plugin This PR includes a new plugin. support:community This issue/PR relates to code supported by the Ansible community. support:core This issue/PR relates to code supported by the Ansible Engineering Team. test This PR relates to tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants