Fix awsservicemap download, added more logging to all commands/sdk calls by sethsec · Pull Request #109 · BishopFox/cloudfox

AWS recently changed the format of https://api.regional-table.region-services.aws.a2z.com/index.json. Cloudfox uses that file, via awsservicemap to download all supported regions and services so that it avoids making calls to service endpoints when that service doesn't exist in that region. But, this dependency made it so that once the JSON format changed, cloudfox was dead in the water. It would always return 0 regions and would skip all of the checks.

FWIW, the json file used to look like this:

It used to look like this:

{
    "aws:region":"ap-east-1",
    "aws:serviceName":"Amazon Translate",
    "aws:serviceUrl":"https://aws.amazon.com/translate/"},
    "id":"translate:ap-east-1"}

The ID used to be service:region

and now, (I think this changed within the last week or two), it now looks like this:

 {
      "id": "1ec05811574987f6ed3e4ecc5d390468760c0aa7007e4c8dbc24a770625cb4e7:ca-central-1",
      "attributes": {
        "aws:region": "ca-central-1",
        "aws:serviceName": "AWS Amplify",
        "aws:serviceUrl": "https://aws.amazon.com/amplify/"
      }
    },

The ID is now sha265:region.

This PR uses the newer version of awsservicemap, which correctly parses the JSON. While I was at it, I fixed a bunch of logging issues. I also got inspired to create cloudfox-info.log which will now log every successfully AWS API call via SDK, which I think will be helpful. This accounts for most of the changes to aws/sdk/*.go files. Lastly, fixed cloudfox-error.log to record more things.

Additional Context

I'm pretty sure this means that any version of cloudfox below v1.17.0 will just not work and will never work again, as they all use the old version of awsservicemap. Everyone that uses cloudfox will have to upgrade to version v1.17.0 once it is released.