17 June 2021 The Lambda Hunter

Finding Deprecated Lambda Runtimes

Chris Allison

Senior SRE Engineer

16 June 2021 - 3 min read

Finding deprecated lambda runtimes in an Organisation

This year AWS have deprecated a number of lambda runtimes:

  • python 2.7
  • nodejs 4.x
  • nodejs 6.x
  • nodejs 8.x
  • nodejs 10.x
  • ruby 2.5

Trusted Advisor has gained a few teeth and now reports the use of any of those runtimes, however, it can’t tell you exactly when the errant function was last used and meta information such as account and region is difficult to surface.

Using a few API calls to AWS we can easily list all lambda functions, sorting them however we choose to. It is still challenging to obtain the last invocation time, but not impossible.

Obtain All Lambda Functions

def getLambdas():
    lm = boto3.client("lambda")
    funcs = []
    kwargs = {}
    try:
        while True:
            lfuncs = lm.list_functions(**kwargs)
            funcs.extend(lfuncs["Functions"])
            if "NextMarker" in lfuncs:
                kwargs["Marker"] = lfuncs["NextMarker"]
            else:
                break
    except botocore.exceptions.ClientError as e:
        # print(f"not enabled (clienterror): {e}")
        pass
    except botocore.exceptions.UnauthorizedOperation as e:
        # print(f"not enabled (unauthorised): {e}")
        pass
    return funcs

Obtain Last Invocation (method A)

See if there is a log group with the same name as the lambda function and find the date of the last stream in that group if it exists.

def getLatestLogTime(lname):
    latest = -1
    cw = boto3.client("logs")
    kwargs = {"logGroupNamePrefix": f"/aws/lambda/{lname}"}
    groups = []
    while True:
        lgroups = cw.describe_log_groups(**kwargs)
        groups.extend(lgroups["logGroups"])
        if "nextToken" in lgroups:
            kwargs["nextToken"] = lgroups["nextToken"]
        else:
            break
    if len(groups) > 0:
        kwargs = {
            "logGroupName": groups[0]["logGroupName"],
            "orderBy": "LastEventTime",
        }
        lstreams = cw.describe_log_streams(**kwargs)
        # print(f"streams: {lstreams}")
        if len(lstreams["logStreams"]) > 0:
            for log in lstreams["logStreams"]:
                if "lastEventTimestamp" in log:
                    xlatest = int(log["lastEventTimestamp"] / 1000)
                    if xlatest > latest:
                        latest = xlatest
        else:
            print(f"""no streams found for loggroup {groups[0]["logGroupName"]}""")
    else:
        print(f"No log group found for /aws/lambda/{lname}")
    return latest

The getLatestLogTime() function above returns a normal, UNIX timestamp (32-bit, 10 digits). The lastEventTimestamp attribute to the logStream is in micro seconds, hence the / 1000 and the cast to an int()

Obtain Last Invocation (method B)

Query the Cloudwatch metrics for the last invocation time of the named lambda function (if there is one). This method is very slow and will not be coded out here as it isn’t really suitable for checking more that 10 - 20 lambda functions at once, meaning that the worker Lambda that is doing the checking is likely to exceed it’s runtime budget.

To investigate this further have a read of the boto3 docs and the boto3 client for cloudwatch.

Reporting

Now we have a list of lambda functions and the capability of finding their last invocation times. We want to weed out and report on all the deprecated lambda runtimes. To do that we’ll iterate through each lambda and build data for a DynamoDB table. We’ll also make a note of the current runtimes and send all the information to our monitoring solution - in our case Wavefront.

We can hold the list of runtimes to be reported upon as an environment string, which’ll make it easy to update in the future when more runtimes are deprecated. I’ve shown it below as a normal string and commented out the call to retrieve it from the environment.

def getDeprecatedData():
    # get a list of all lambda functions
    funcs = getLambdas()

    # make a list of the deprecated runtimes
    # runtimes = os.environ.get("RUNTIMES", "").split(",")
    runtimes = "nodejs6.10,nodejs8.10,python2.7,ruby2.5,nodejs10.x".split(",")

    # create a dict to allow us to count all the different runtimes
    allruntimes = {}

    # list to return the list of deprecated lambdas
    opfuncs = []

    # iterate through each lambda
    for func in funcs:
        # have we seen an instance of this runtime before?
        # if not create the counter
        if func["Runtime"] not in allruntimes:
            allruntimes[func["Runtime"]] = 0

        # iterate the runtime counter for this particular runtime
        allruntimes[func["Runtime"]] += 1

        # check if this lambda's runtime is deprecated
        if func["Runtime"] in runtimes:

            # obtain the last invocation time (if possible)
            func["lastlogtime"] = getLatestLogTime(func["FunctionName"])

            # add the function to the return list
            opfuncs.append(func)

    # send the metrics to the wavefront monitoring system
    # partioned by organisation, account, region and runtime
    for rt in allruntimes:
        mname = f"{orgid}.{acctid}.{region}.{rt}"
        mval = allruntimes[rt]
        sendMetric(mname, mval)

    # return the list of deprecated runtimes
    return opfuncs

I won’t go into the mechanics of writing data to DynamoDB here, nor creating spreadsheets from that data. I’ll create a seperate post for that as it can be ‘tricky’.

Monitoring

Ultimately, our aim is to have good visibility of the state of our AWS estate. We use Wavefront extensively for this. The screenshot below shows the result of collecting information regarding Lambda Functions and how it can be displayed in an easily digestible format. I’ve massaged the figures somewhat to show what is possible - this does not represent any part of Centrica’s estate.